Assessing the current state of a physical area of a healthcare facility using image analysis

ABSTRACT

Techniques for determining issues associated with a current state of an area of a healthcare facility are provided. In one example, a computer-implemented method comprises identifying an area of a healthcare facility included in image data captured of the healthcare facility, and characterizing aspects of a current state of the area of the area based on analysis of the image data, including determining object information characterizing objects included in the area. The method further comprises determining whether one or more of the aspects of the current state of the area deviate from a defined requirement for the area based on evaluation of the object information. Based on a determination that an aspect of the current state of the area deviates from the defined requirement for the area, the method can further comprise generating feedback information regarding the deviating aspect of the current state of the area.

RELATED APPLICATION

This application is a continuation of and claims priority to U.S. patentapplication Ser. No. 15/442,386, filed Feb. 24, 2017 and titled“PROVIDING AUXILIARY INFORMATION REGARDING HEALTHCARE PROCEDURE ANDSYSTEM PERFORMANCE USING AUGMENTED REALITY,” the entirety of whichapplication is incorporated herein by reference.

TECHNICAL FIELD

The subject matter disclosed herein relates to techniques for assessingthe current state of a physical area of a healthcare facility usingimage analysis.

BACKGROUND

In the healthcare field, there are often best practices for performinghealthcare related procedures, whether it is a medical procedureperformed by a healthcare clinician on a patient, a clerical procedureinvolving data entry, a maintenance procedure involving ensuring amedical environment and associated equipment is properly maintained, orthe like. These best practices can be government mandated or implementedby the medical organization's internal policies, and are intended tofacilitate optimal performance of the healthcare organization in termsof quality of patient care and financial return. Further, certainactivity may have different official procedures based on whichgovernment or insurance company is paying for the healthcare activity.

Given the complexity of modern healthcare organizations attributed tothe medical services provided and the highly variable and sensitivenature of patient care, it can be extremely difficult for healthcareprofessionals to adhere to such best practices. For example,administering medical procedures to a patient traditionally relies onmany healthcare personnel interacting with one another in highlyvariable and volatile scenarios. The ability for each of individual toadhere to best practice standards can be influenced by the actions ofeveryone involved, the condition of the patient, the stress and fatigueof the clinicians, the availability and proper functioning of necessarysupplies and equipment, and possible distractions in the surroundings.Human error is inevitable. In the healthcare field, the costs associatedwith human errors can have a severe impact on patient well being as wellas financial cost to the serving healthcare organization. Further, manyinsurance companies only pay for certain procedures if the procedurehave been performed according to their set forth protocol. Thusmechanisms for facilitating adherence to defined practice standards andminimizing errors associated with deviations from the defined practicestandards are of great importance.

SUMMARY

The following presents a summary to provide a basic understanding of oneor more embodiments of the invention. This summary is not intended toidentify key or critical elements, or delineate any scope of theparticular embodiments or any scope of the claims. Its sole purpose isto present concepts in a simplified form as a prelude to the moredetailed description that is presented later. In one or more embodimentsdescribed herein, systems, computer-implemented methods, apparatusand/or computer program products that facilitate providing auxiliaryinformation regarding healthcare procedure and system performance usingaugmented reality.

In one embodiment, a system is provided that comprises a memory thatstores computer executable components, and a processor that executescomputer executable components stored in the memory. The computerexecutable components can comprise a procedure characterizationcomponent configured to determine descriptive information regardingperformance of a healthcare related procedure by a user based on inputdata generated in real-time during the performance of a healthcareprocedure by the user, wherein the input data comprises video capturedof the user during the performance of the healthcare related procedure.The computer executable components can further comprise a procedureassessment component configured to determine whether an aspect of theperformance of the healthcare related procedure currently beingperformed by the user deviates from a defined protocol for thehealthcare related procedure based on comparison of the descriptiveinformation with reference descriptive parameters for the healthcarerelated procedure, and a feedback component configured to determinefeedback information regarding correction of the aspect in response to adetermination that the aspect deviates from the defined protocol. Insome implementations, the computer executable components can alsocomprise an augmented reality data component configured to generateoverlay data representative of the feedback information for projectionon a display over a current view of the user in response to thedetermination that the aspect deviates from the defined protocol.

In another embodiment, a system is provided that comprises a memory thatstores computer executable components, and a processor that executescomputer executable components stored in the memory. The computerexecutable components can comprise a recognition component that analyzescaptured image data of a physical area of a healthcare facility andidentifies one or more objects within the captured image data, and anassessment component that assesses a state of performance of thehealthcare facility based on the one or more of the objects. Thecomputer executable components can further comprise a feedback componentthat determines feedback information regarding the state of performance,a selection component that selects a subset of the feedback informationfor providing to a user based in part on a context of the user, and anaugmented reality data component that generates overlay datarepresentative of the subset of the feedback information for projectionon a display over a current view of the physical area of the healthcarefacility that is viewed by the user.

In one or more additional embodiments, a method is described thatcomprises receiving, by a system comprising a processor, input datagenerated in real-time during performance of a healthcare procedure by auser, wherein the input data comprises video captured of the user duringthe performance of the healthcare related procedure. The method furthercomprises determining, by the system, descriptive informationcharacterizing the performance based on the input data, wherein thedescriptive information characterizes at least actions performed by theuser during the procedure. The method further comprises determining, bythe system, whether an aspect of one of the actions currently beingperformed by the user deviates from a defined protocol for thehealthcare related procedure based on comparison of the descriptiveinformation with reference descriptive parameters for the healthcarerelated procedure, and determining, by the system, feedback informationregarding correction of the aspect in response to a determination thatthe aspect deviates from the defined protocol. In some implementations,the method can also comprise generating, by the system, overlay datarepresentative of the feedback information for projection on a displayover a current view of the user in response to the determination thatthe aspect deviates from the defined protocol.

DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a block diagram of an example system that facilitatesproviding auxiliary information regarding healthcare procedureperformance using augmented reality (AR) in accordance with one or moreembodiments described herein.

FIGS. 2-4 present example AR visualizations including auxiliaryinformation presented to a clinician regarding current performance of ahealthcare procedure by the clinician in accordance with one or moreembodiments described herein.

FIG. 5 provides a flow diagram of an example method for providingauxiliary information regarding healthcare procedure performance usingAR in accordance with one or more embodiments described herein.

FIG. 6 illustrates a block diagram of an example system that facilitatesproviding auxiliary information regarding healthcare procedureperformance using AR in accordance with one or more embodimentsdescribed herein.

FIG. 7 illustrates a block diagram of an example system that facilitatesproviding auxiliary information regarding healthcare procedureperformance using AR in accordance with one or more embodimentsdescribed herein.

FIG. 8 illustrates a block diagram of an example user device thatfacilitates providing auxiliary information regarding healthcareprocedure and system performance using AR in accordance with one or moreembodiments described herein.

FIG. 9 illustrates a block diagram of an example user device thatfacilitates providing auxiliary information regarding healthcareprocedure and system performance using AR in accordance with one or moreembodiments described herein.

FIG. 10 illustrates a block diagram of an example system thatfacilitates providing auxiliary information regarding healthcare systemperformance using AR in accordance with one or more embodimentsdescribed herein.

FIGS. 11 presents an example AR visualization including auxiliaryinformation regarding usage and/or performance of a healthcare systemequipment in accordance with one or more embodiments described herein.

FIGS. 12A-12B presents example AR visualization respectively includingauxiliary information regarding usage and/or performance of a healthcaresystem equipment in accordance with one or more embodiments describedherein.

FIG. 13 illustrates a block diagram of an example system thatfacilitates providing auxiliary information regarding healthcare systemperformance using AR in accordance with one or more embodimentsdescribed herein.

FIG. 14 presents an example AR visualization including auxiliaryinformation regarding employees of a healthcare organization inaccordance with one or more embodiments described herein.

FIG. 15 presents an example AR visualization including auxiliaryinformation regarding various aspects of an operating room environmentin accordance with one or more embodiments described herein.

FIG. 16 illustrates a block diagram of an example system thatfacilitates providing auxiliary information regarding healthcare systemperformance using AR in accordance with one or more embodimentsdescribed herein.

FIG. 17 illustrates a flow diagram of an example method that facilitatesproviding auxiliary information regarding healthcare procedureperformance using AR in accordance with one or more embodimentsdescribed herein.

FIG. 18 illustrates a flow diagram of another example method thatfacilitates providing auxiliary information regarding healthcareprocedure performance using AR in accordance with one or moreembodiments described herein.

FIG. 19 illustrates a flow diagram of an example method that facilitatesproviding auxiliary information regarding healthcare system performanceusing AR in accordance with one or more embodiments described herein.

FIG. 20 illustrates a block diagram of an example non-limiting operatingenvironment in which one or more embodiments described herein can befacilitated.

DETAILED DESCRIPTION

The following detailed description is merely illustrative and is notintended to limit embodiments and/or application or uses of embodiments.Furthermore, there is no intention to be bound by any expressed orimplied information presented in the preceding Background or Summarysections, or in the Detailed Description section.

One or more embodiments of the subject disclosure are directed tocomputer processing systems, computer-implemented methods, apparatusand/or computer program products that facilitate preventing orminimizing human error associated with performance of healthcare relatedprocedures by a healthcare professional providing auxiliary informationregarding healthcare procedure performance using augmented reality (AR).In particular, one or more embodiments of the disclosed systems providemechanisms to improve human behavior through AR guidance. For example,the subject AR systems and methods contemplated herein can help guidestaff to complete the correct procedure steps based on which procedureis correct based on many factors discussed herein, and including whichgovernment and/or insurance agency may be paying for the activity.Further, in some implementations, the subject AR systems can detect whenhuman behavior may be an improvement over current procedures and methodsand propose those improvements to leadership.

The term “healthcare professional” is used herein to refer to any entitythat performs a healthcare related procedure or action. The type of thehealthcare related procedure or action can vary. For example, ahealthcare related procedure can include but is not limited to, amedical procedure performed by a healthcare clinician on a patient, aclerical procedure performed by a clerical employee, a maintenanceprocedure performed by a maintenance employee, or the like. Thus ahealthcare professional can include for example but not limited to: aclinician (e.g., a physician, a nurse, a physician's assistant, atherapist, a medical technician, etc.), a clerical worker of ahealthcare organization (e.g., a medical records technician, a medicalsecretary, a medical billing assistant), a maintenance worker of ahealthcare organization (e.g., a medical equipment technician), or thelike. A medical procedure is a course of action intended to achieve aresult in the delivery of healthcare. A medical procedure with theintention of determining, measuring, or diagnosing a patient conditionor parameter is also called a medical test. Other common kinds ofmedical procedures are therapeutic (e.g., intended to treat, cure, orrestore function or structure), such as surgical and physicalrehabilitation procedures. In regard to the subject disclosure, amedical procedure may involve some physical interaction between aclinician and a patient. Medical procedures are often defined by astandardized coding system wherein the respective procedures havedefined codes. The coding system can vary from one medical organizationto another or may be standardized by a governing body.

In one or more implementations, as a healthcare professional isperforming a healthcare related procedure, a computer system can monitorthe respective actions performed by the healthcare professional inreal-time. For example, in some embodiments, image data (e.g., videoand/or still images) and audio data can be captured of the healthcareprofessional performing the procedure. With these embodiments, thecomputer system can receive the image and audio data in real-time (e.g.,as it is being captured). Using image based pattern recognitiontechniques (e.g., object detection, movement/motion detection, etc.),the computer system can determine respective aspects of the procedurebeing performed in real-time. For example, based on analysis of theimage data, the computer system can identify a specific type ofprocedure being performed by the healthcare professional (e.g., aspecific type of medical procedure performed on a patient, a specifictype of clerical procedure, a specific type of maintenance procedure,etc.). The computer system can further identify respective actionsperformed by the healthcare professional in association with performanceof the procedure and characteristics of the respective actions. Forexample, in association with monitoring a clinician performing anintravenous (IV) cannulation procedure, the computer system can identifyand characterize steps of the procedure including but not limited to:preparation of the IV site, preparation of the tubing and the fluid bag,application of the tourniquet, selection of the catheter, insertion ofthe catheter, removal of the tourniquet, attaching the tubing, andopening of the IV line. According to this example, using automatedpattern recognition image analysis techniques, the computer system canidentify if and when the healthcare professional is performing therespective steps of the IV cannulation procedure, timing of performanceof the respective steps, order of performance of the respective steps,and various defined characteristics of the respective steps (e.g.,location of the IV site, size of the tubing, size of the catheter,dosage of the fluid bag, location of the tourniquet, etc.).

In some embodiments, the computer system can also receive various otherforms of input/feedback that can be used to monitor and characterizevarious aspects of a procedure being performed by a healthcareprofessional. For example, these other forms of input/feedback caninclude audio spoken by the healthcare professional, the patient, otherhealthcare professionals involved in the procedure and/or audiogenerated by a machine associated with the procedure (e.g., an alarmsound). In another example, the feedback can include sensory feedbackcaptured via motion and/or biometric sensors worn by the healthcareprofessional, the patient, or other healthcare professionals involved inthe procedure. In yet another example, the feedback can include motiondata regarding motion of a medical instrument or device used during theprocedure. In yet another example, the feedback can include inputreceived or captured by a machine or instrument used during theprocedure.

In association with identifying and characterizing the respective stepsof the procedure performed by the healthcare based on the various typesof input/feedback discussed above, the computer system can furtheranalyze the steps to determine whether the healthcare professional isperforming the procedure correctly and in accordance with predefinedbest practice standards for the procedure. In some implementations, ifthe computer system determines the healthcare professional is performingan aspect of the procedure incorrectly or otherwise in deviation of thepredefined standards for the procedure, the computer system can providethe healthcare professional with auxiliary information in real-timeregarding the error. For example, the computer system can provide thehealthcare professional with auxiliary information identifying theerror. In another example, the computer system can provide thehealthcare professional with auxiliary information indicating how tocorrect the error. In other implementations, if the computer systemdetermines the healthcare professional is performing an aspect of theprocedure correctly, the computer system can provide the healthcareprofessional with auxiliary information in real-time acknowledging orpraising the healthcare professional's good performance. In yet anotherimplementation, based on recognition of a type of procedure beingperformed by a healthcare professional, the computer system can monitorthe performance of the procedure and provide auxiliary information withstep by step guidance regarding how to perform the procedure correctly.

In accordance with various embodiments described herein, the auxiliaryinformation can be provided to the healthcare professional in real-timein using an AR device. Augmented reality (AR) is a live direct orindirect view of a physical, real-world environment whose elements areaugmented (or supplemented) by computer-generated sensory input such asgraphics, sound, video, or other data. As used herein, an “AR device”includes a computer device capable of generating an AR experience. Insome embodiments, an AR device can include a device that comprises adisplay configured to render overlay data onto a live view of real-worldphysical environment. For example, in some implementations, the ARdevice is an eyewear device, such as glasses, goggles, a heads-updisplay (HUD), a contact lens or pair of contact lenses, or the like,that is configured to render overlay data onto a display of the eyeweardevice while a user is wearing the eyewear device and also viewing thereal-world environment. In another implementation, the AR device caninclude an eyewear device or a handled computing device (e.g., a tablet,a smartphone, or the like), configured to render overlay data onto liveimage or video data captured of the environment. In otherimplementations, an AR device can include a device configured to renderaudio, vibratory or other sensory signals in association with viewing areal-world environment. The AR device may include speakers or headphonesfor audio output.

In this regard, in one or more embodiments, as a healthcare professionalis performing a healthcare related procedure, the healthcareprofessional can wear, hold, or otherwise view an AR device. The ARdevice can provide a real or live image/video view of the healthcareprofessional's environment from the healthcare professional'sperspective via a display of the AR device. For example, with referenceto the IV cannulation procedure example, a healthcare professionalperforming the IV cannulation procedure can wear or otherwise employ anAR device during the procedure that presents a live view, from thehealthcare professional's perspective, of the IV site, the patient, thefluid bag, etc., as the healthcare professional is viewing these objectswhen performing the procedure. If the system determines that thehealthcare professional is performing or has performed an aspect of theprocedure incorrectly, the system can generate auxiliary informationregarding the error in the form of overlay data to be projected on thedisplay of the AR device over the live view (e.g., either real orvideo/image data) of the environment. For example, the overlay data caninclude but is not limited to: text, a symbol, a marker, a tag, animage, a video, an animation, or a hyperlink. The AR device can furtherrender the overlay data via the display of the AR device over the liveview (e.g., either real or video/image) of the environment alsopresented on or viewed through the display.

In some implementations, depending on the type of overlay data and/orthe aspect of the procedure the overlay data associated with, theoverlay data can be displayed in a manner such that it appears spatiallysituated (e.g., in two-dimensional (2D) or three-dimensional (3D) space)relative to an object or part of an object associated with the aspect ofthe procedure. For example, with reference again to the IV cannulationprocedure example, if the system detects incorrect placement of the IVneedle relative to the patient's IV site, the system can generateoverlay data that indicates the incorrect placement of the IV needle andidentifies how to correct the placement of the IV needle relative to thepatient's IV site. The AR device can further render the overlay datasuch that it appears spatially situated relative to a live view of thepatient's IV site. By providing such visual auxiliary informationregarding healthcare procedure performance in real-time using AR, thedisclosed techniques greatly reduce human error by giving the healthcareprofessional real-time and relevant instructions that guide thehealthcare professional with making necessary adjustments during theprocedure.

One or more additional embodiments of the subject disclosure aredirected to computer processing systems, computer-implemented methods,apparatus and/or computer program products that facilitate providingauxiliary information regarding healthcare system performance inreal-time using AR. With respect to large scale healthcare organizationsproviding various different patient services and having many (e.g.,hundreds, thousands, or more) employees that perform a variety ofinterrelated clinical and non-clinical roles, it can be extremelydifficult for an administrative worker to have a thorough understandingof a current state of performance of the healthcare organization withrespect to different areas of the healthcare organization, let alone thehealthcare organization as whole. A healthcare administrator isgenerally responsible for the overall directing of operations of thehealthcare organization to achieve the long-range and short-range goalsand objectives of the organization. For example, the administrator maybe responsible for the daily administration and leadership of the acertain department or sector of the organization or the organization(e.g., surgery); accountable for directing, coordinating and controllingall aspects of financial management, inventory management, facilitymaintenance, surgical services, patient relations and staff; responsiblefor fostering positive work relationships and serve as a liaison betweenthe governing body, medical staff, employees and all areas of thedepartment; and accountable for improving the clinical and financialoperations of the department. It is often difficult for an administratorto perform these duties merely by looking at data reports with chartsand numbers. For example, the administrator may not have a strongconcept regarding how the different types of equipment, supplies, andclinician services associated with a particular department are utilizedon a regular basis, and the current issues affecting their performanceIn another example, the administrator may not have a good understandingof what a particular piece of equipment is used for, how its use impactsthe workflow of a particular medical sector of the hospital, how its useinfluences other supplies and systems of the healthcare facility, andthe like.

The subject disclosure provides computer processing systems,computer-implemented methods, apparatus and/or computer program productsthat facilitate a user (e.g., an administrator of the healthcareorganization or another suitable entity) with understanding informationconcerning a state of performance of a healthcare organization byproviding the healthcare administrator with auxiliary informationassociated with a current physical area of a healthcare facility of theorganization the healthcare professional is viewing or otherwiseexamining In one or more implementations, the state of performance canrelate to a clinical and/or financial performance state of thehealthcare organization. In particular, based on captured image dataand/or other input information a computing system can determine an areaof a healthcare facility that the user is currently looking at, or hasrequested to look at. The computing system can further identify objects,equipment, supplies, people, etc., included in the area (or associatedwith the area), and further determine auxiliary information associatedwith the objects, equipment, supplies, people, etc.,

The auxiliary information can include but is not limited to, equipmentinformation regarding utilization and/or performance of equipmentassociated with the area, employee information regarding utilizationand/or performance of the employees associated with the area, andrecommendation information regarding recommendations for improving theperformance of the healthcare organization in association with changingan aspect of the equipment and/or employee utilization and/orperformance associated with the area (e.g., addition or removal ofresources). For example, in association with viewing a particular roomin a hospital, the user can be automatically provided with auxiliaryinformation associated with performance of the healthcare organizationbased on utilization of the room, including but not limited to:information identifying medical supplies and equipment included in theroom, information regarding usage of the medical supplies and equipment,and financial information concerning usage of the medical supplies andequipment. In another implementation, the auxiliary information caninclude employee information regarding employees included in the room orotherwise associated with the room. For example, the employeeinformation can but is not limited to: identities of employees includedin or otherwise associated with the room (e.g., employees that use theroom, employees scheduled for procedures in the room, employeesresponsible for maintenance of the room and the like), role descriptioninformation describing one or more aspects of the role or serviceprovided the employee, performance information regarding performance ofthe employee with respect to their role or job duties, financialinformation regarding ROI associated with the respective employees,personal information associated with the respective employees, and thelike. The auxiliary information can further include various other typesof information associated with utilization of the room that relates toor have an impact on the clinical and financial performance of thehealthcare organization, such as but not limited to: proceduresperformed in the room, frequency of the procedures, outcomes of theprocedures, ROI of the procedures, maintenance issues associated withthe room, scheduled updates to the room, purchase orders, and the like.

In various implementations, the amount and type of auxiliary informationthat is provided to a particular administrator can be filtered andselected based on a current context of the user. For example, theauxiliary information that is provided to a maintenance worker canreflect maintenance issues associated with equipment in an observed areawhile the auxiliary information that is provided to a finance executivecan concern cost and ROI performance data associated with equipmentand/or medical services supported by the area. In another example, inassociation with providing a user with information regarding employeesthe user interacts with, the auxiliary information provided can be basedon a relationship between the user and the employee and/or an intentionfor interacting with the employee. For example, if the user is merelypassing by an employee that the user is not well acquainted with, theuser can be provided with information reminding the user of theemployee's name and prompting the user to congratulate the employee onhis or her promotion. Alternatively, if the user is meeting with anemployee to discuss improving the employee's performance, the user canbe provided with auxiliary information that highlights some importantpast performance statistics.

Similar to the auxiliary information provided to user in associationwith performance of a procedure, in some implementations the auxiliaryinformation regarding the state of performance of the healthcareorganization can be provided to the user using an AR experience. Forexample, using an AR device, a user can physically visit an area of ahealthcare facility and receive auxiliary information concerning theperformance of the area, including equipment and supplies associatedwith the area, people (e.g., employees, patients, visitors, etc.)associated with the area, and the like. The auxiliary information can berendered to the user via the display of the AR device as overlay datathat is overlaid onto a current view of the area viewed through or onthe display. In another implementation, the user can remotely view thearea by viewing video and/or image data captured of the area (e.g., viaone or more cameras dispersed throughout the healthcare facility).According to this implementation, the auxiliary data can be overlaidonto the video and/or image data of the area that is displayed on theuser's remote device. Still in yet another implementation, the user canview the area using a virtual reality (VR) system/device. VR is arealistic and immersive simulation of a three-dimensional environment,created using interactive software and hardware, and experienced orcontrolled by movement of the body. According to this implementation,using a VR device, a user can be presented with a 3D spacemodel/visualization of the area that appears to the user as if the useris actually standing in the 3D space model. The auxiliary data regardingthe area can be projected onto or otherwise integrated within the 3Dspace model view of the area.

In various embodiments, regardless as to whether the user is viewing theauxiliary data using an AR device or a VR device, the auxiliary data canbe spatially aligned with respective objects, people,equipment/supplies, etc., that is associated with. For example,auxiliary data concerning how a supply cabinet in a room is understocked with certain supplies can be overlaid on a direct view of thesupply cabinet (e.g., the supply cabinet viewed through a transparentdisplay, such as AR glasses or goggles), or an indirect view of thesupply cabinet (e.g., video/image data or 3D model data of the supplycabinet presented on a display).

The various embodiments are now described with reference to thedrawings, wherein like referenced numerals are used to refer to likeelements throughout. In the following description, for purposes ofexplanation, numerous specific details are set forth in order to providea more thorough understanding of the one or more embodiments. It isevident, however, in various cases, that the one or more embodiments canbe practiced without these specific details.

FIG. 1 illustrates a block diagram of an example, non-limiting system100 that facilitates providing auxiliary information regardinghealthcare procedure performance using AR in accordance with one or moreembodiments described herein. Aspects of systems (e.g., system 100 andthe like), apparatuses or processes explained in this disclosure canconstitute machine-executable component(s) embodied within machine(s),e.g., embodied in one or more computer readable mediums (or media)associated with one or more machines. Such component(s), when executedby the one or more machines, e.g., computer(s), computing device(s),virtual machine(s), etc. can cause the machine(s) to perform theoperations described.

System 100 includes a user device 102, a server device 108 and one ormore external information sources 138. The respective devices/sources ofsystem 100 can include computer executable hardware and/or softwarecomponents. For example, in the embodiment shown, the server device 108includes (or is otherwise operatively coupled to) at least one memory130 that stores computer-executable components. In one or moreembodiments, these computer executable components include the augmentedreality (AR) assistance module 110 and associated components. The serverdevice 108 can also include (or is otherwise operatively coupled to) atleast one processor 134 that executes the computer-executable componentsstored in the memory 130. The server device 108 can further include asystem bus 136 that can couple the various components of the serverdevice 108 including, but not limited to, the AR assistance module 110,the memory 130 and the processor 134.

The AR assistance module 110 can be configured to perform various ARtype services associated with facilitating healthcare professionalsperforming their jobs. In accordance with system 100, the AR assistancemodule 110 is particularly configured to assist healthcare professionalswhen performing a defined procedure by providing the healthcareprofessional with auxiliary information, determined in real-time, thatis relevant to a current aspect of the procedure being performed. Invarious implementations, the auxiliary information is designed to helpthe healthcare professional perform the procedure in accordance with apredefined protocol, standard, or best practice for performing theprocedure. The auxiliary information can be provided to the healthcareprofessional in real-time during performance of the procedure via a userdevice 102 associated with the healthcare professional.

For example, in one or more embodiments, the user device 102 is an ARdevice that is worn by (e.g., an eyewear device) or otherwise viewed bythe healthcare professional during performance of the procedure.According to these embodiments, the auxiliary information can berendered via a display (e.g., display 106) of the AR device in the formof overlay data that is overlaid or projected onto a live view of thehealthcare professional's environment. For example, the live view caninclude a direct view of the healthcare professional's environment asviewed through a transparent display (e.g., display 106), such as atransparent display that is worn over the healthcare professionals eyeor eyes (e.g., glasses, goggles, contact lenses, etc.). In anotherexample, the live view can include an indirect view of the healthcareprofessional's environment that comprises video or image data of theenvironment captured from the healthcare professional's perspective viaa camera (e.g., camera 104) and rendered on a display (e.g., display106) of the AR device.

For example, in the embodiment shown, the user device 102 includes atleast one camera 104 and a display 106. Speakers or other audio outputmay also be included in some embodiments. The camera 104 can include anysuitable camera configured to capture image data, including still imagesand/or video. For example, in one or more embodiments, the camera 104can capture live video of a healthcare professional's environment inassociation with performance of a healthcare procedure by the healthcareprofessional. The live video can be rendered via the display 106 and/orprovided to the server device 108 for processing by the AR assistancemodule in association with monitoring the healthcare professional'sperformance of the healthcare procedure and providing real-time feedbackto the healthcare professional regarding the performance In someimplementations, the camera 104 can be configured to capture and/ordetermine depth or distance information for features included in thecaptured image data. The depth information can be used to facilitatespatially aligning overlay data with objects appearing in the image dataand/or a real physical view of the healthcare professional'senvironment. The spatial data can also be used to facilitate identifyingobjects appearing in the image data, spatial relationships of objectsappearing in the image data, and tracking movement of objects appearingin sequence of images captured over time (i.e., video data).

In other embodiments, one or more additional cameras external to theuser device 102 can be dispersed throughout a healthcare environment andcapture image data (e.g., still images and/or video) of a healthcareprofessional performing a healthcare procedure. For example, one or morecameras can be located at different positions in an operating room,examination room, laboratory room, at different positions throughout ahospital, etc. These cameras can be configured to capture image dataand/or video data associated with procedures performed in the healthcareenvironment and provide the image data to the server device 108 inreal-time for processing by the AR assistance module 110.

The one or more external information sources 138 can include varioussystems and databases that can provide information to the AR assistancemodule 110 that facilitates evaluating performance of a healthcarerelated procedure and/or performance of various aspects of a healthcareorganization. For example, the one or more external information sources138 can include systems, servers, devices, etc., that are internal to aparticular healthcare organization providing and/or controlling theserver device 102 and associated AR assistance module 110, as well asvarious remote systems, server, devices, etc., accessible via a network(e.g., the Internet). The one or more external information sources 138can also include various systems and databases that facilitate the ARassistance module 110 with respect to generating and providing auxiliaryinformation to healthcare professional, in real-time, based on theperformance evaluations.

For example, the one or more external information sources 138 caninclude a source with information regarding various defined healthcarerelated procedures that can be monitored by the AR assistance module110, including information identifying the respective procedures anddescriptive information comprising descriptive parameters describingcharacteristics of the respective procedures. In another example, theone or more external information sources 138 can include informationregarding defined guidelines or protocols for the respective healthcarerelated procedures. These guidelines/protocols can be defined by thehealthcare organization and/or government mandated. For example,government organizations like the U.S. institutions of NationalInstitutes of Health or Medicare may be examples of institutions withmandated procedures in certain cases.

In some implementations, these guidelines and protocols can be optimizedover time using various machine learning techniques and data monitoredfor repeated or similar procedures. For example, the procedureguideline/protocol information can include descriptive parameters thatdefine required steps or actions for a procedure and timing for therespective steps or actions, including an order of for performance ofthe respective actions. The procedure guideline/protocol information canalso include descriptive parameters that define how the respectiveactions/steps should be performed, what supplies/instruments should beused, how to dispose of supplies, how to interact with a patient and/orother healthcare professionals during a procedure and the like. Inanother example, the guideline/protocol information can includedescriptive parameters regarding responsive actions/steps to be takenwhen various potential events, reactions, or complications arise duringa procedure. It should be appreciated that the various types ofinformation included in guideline/protocol information regarding how toperform a medical related procedure according to a desired standard canvary depending on the type of procedure performed and the organizationdefining the guideline/protocol. In the embodiment shown, such procedureguideline/protocol information is located at the server device 108(e.g., in memory 130 as procedure guideline/protocol information 132),however, it should be appreciated that the specific location ofinformation can vary so long as it is accessible to the AR assistancemodule to facilitate real-time evaluation of procedures being performedby healthcare professionals.

In another example, the one or more external information sources 138 caninclude employee information identifying all healthcare professionalemployees of a healthcare organization that employs the AR assistancemodule 110 to facilitate monitoring and improving the performance of thehealthcare organization. Such healthcare organizations can include butare not limited to: hospital networks, hospitals, nursing homes,assisted living facilities, physical therapy organizations, billingorganizations, pharmacies, etc. For example, the employee informationcan identify the respective healthcare professional employees, theirroles (e.g., job title), the patients they work with (if having aclinician role), the procedures they are authorized to perform, theprocedures they are expected to perform, and the like. The employeeinformation can also include any possible performance information thatcan be monitored and recorded for the employee over time, such as hoursworked, procedures performed, outcomes of the procedures, evaluations ofthe procedures, satisfaction of patients, reviews of the employee,information regarding how the employee performs certain tasks relativeto other employees or another benchmark, and the like. The employeeinformation can also include financial information regarding salary ofthe respective employees, return on investment (ROI) generated from therespective employees, and the like. The employee information can alsoinclude personal information associated with the respective employees,such as information regarding an employee's family and friends,preferences, birthday, hobbies, awards, vacations, and the like. Theemployee information can be updated on a regular basis (e.g., hourly,daily, weekly, etc.) such that the information source provides currentinformation regarding various aspects of an employee's work life andpersonal life.

The one or more external information sources 138 can also includepatient record information, including information identifying patientsof a healthcare organization or healthcare provider that employs the ARassistance module 110 to facilitate monitoring and improving theperformance of the healthcare organization or healthcare provider. Thepatient information can include but is not limited to: patient healthrecords (e.g. electronica medical records or “EMRs”), patient laboratoryresult information, patient imaging study information, patient diagnosisinformation, scheduling information, physician information, prescriptioninformation, care plan information, allergy information, insuranceinformation, and the like. The patient information can also includepersonal information associated with the patient, such as informationregarding the patient's family and friends, preferences, birthday,hobbies, and the like.

The one or more external information sources 138 can also includeinformation regarding the physical environment of a healthcareorganization facility, such as but not limited to: information regardingthe layout or floor plan of the facility, information identifyingrelative locations of the respective buildings, sectors, rooms, etc.,information identifying what the respective buildings, sectors, rooms,etc., are used for, information regarding equipment at the healthcarefacility and relative physical locations of the equipment, informationregarding medical supplies at the healthcare facility and relativelocations of the medical supplies, and the like. The one or moreexternal information sources 138 can also include information regardingoperating status and maintenance issues associated with the equipment.The one or more external information sources 138 can also includeperformance information associated with services rendered, quality ofpatient, and financial performances of the healthcare organization basedon specific areas and equipment of the facility. For example, suchperformance information can relate to utilization of the equipment andsupplies including financial information associated with the utilizationof the equipment and supplies, procedures performed in the respectivebuildings, sectors, rooms, etc., outcomes of the procedures, quality ofthe procedures, frequency of the procedures, durations of the proceduresand financial information associated with the procedures. Suchperformance information can include information regarding relationshipsbetween patients, patient medical conditions, patient outcomes, patientquality, aspects of courses of patient care (e.g., diagnoses, inpatientand outpatient length of stay, operations/procedures performed), medicalprocedures, medical personnel, resources employed, pharmaceuticalsemployed, medical devices employed, costs, reimbursement, return oninvestments (ROI), and the like.

In the embodiment shown, the user device 102 and the server device 108are configured to operate in a server/client relationship, wherein theserver device 108 employs the AR assistance module to monitor andevaluate a medical related procedure being performed by a healthcareprofessional and provide auxiliary information regarding the procedureto the healthcare professional at the user device 102. In someembodiments, the user device 102 can provide the server device 108 withimage/video data captured of the healthcare professional performing theprocedure, (e.g., via the camera 104), that is used by the AR assistancemodule 110 to evaluate the performance In other implementations, theimage/video data can be captured by and provided to the server device108 via one or more other cameras in the located in the healthcareprofessional's environment that are configured to capture video/imagedata of the healthcare professional performing a medical relatedprocedure. However, it should be appreciated that although system 100employs a server/client architecture, the various features of thesubject disclosure are not limited to this architecture. For example, insome embodiments, the AR assistance module 110 and/or one or morecomponents of the AR assistance module 110 can be provided at the userdevice 102 or another device (not shown). In another example, the one ormore external information sources 138 can be located at the user device102 and/or the server device 108.

In one or more embodiments, the user device 102, the server device 108and/or the one or more external information sources 138 can becommunicatively coupled either directly or indirectly via onecommunication networks (not shown). Such communication networks caninclude wired and wireless networks, including, but not limited to, acellular network, a wide area network (WAN) (e.g., the Internet) or alocal area network (LAN). For example, the user device 102 can beconfigured to communicated with the server device 108 (and vice versa)using virtually any desired wired or wireless technology, including butnot limited to: wireless fidelity (Wi-Fi), global system for mobilecommunications (GSM), universal mobile telecommunications system (UMTS),worldwide interoperability for microwave access (WiMAX), enhancedgeneral packet radio service (enhanced GPRS), third generationpartnership project (3GPP) long term evolution (LTE), third generationpartnership project 2 (3GPP2) ultra mobile broadband (UMB), high speedpacket access (HSPA), Zigbee and other 802.XX wireless technologiesand/or legacy telecommunication technologies, BLUETOOTH®, SessionInitiation Protocol (SIP), ZIGBEE®, RF4CE protocol, WirelessHARTprotocol, 6LoWPAN (IPv6 over Low power Wireless Area Networks), Z-Wave,and/or an ultra-wideband (UWB) standard protocol.

The user device 102 can include any suitable computing device associatedwith a user and that can receive and/or render auxiliary informationgenerated by the AR assistance module 110. For example, the one userdevice can include a desktop computer, a laptop computer, a television,an Internet enabled television, a mobile phone, a smartphone, a tabletuser computer (PC), or a personal digital assistant (PDA). In variousexemplary embodiments, the user device 102 includes a wearable computingdevice configured to provide an AR, virtual reality (VR) and/or mixedreality experience in association with rendering auxiliary dataassociated with performance of a medical related procedure by ahealthcare professional. For example, the user device 102 can includedevice configured to be worn on or near a user's eye, such as glasses,goggles, contact lenses, a HUD device, and the like. As used in thisdisclosure, the terms “user,” “healthcare professional,” “clinician,”“employee,” “administrator” and the like refer to a person, entity,system, or combination thereof that can employ system 100 (or additionalsystems described in this disclosure) using a user device 102.

In the embodiment shown, the AR assistance module 110 includes procedureidentification component 112, procedure assessment component 116,context component 118, feedback component 120, and augmented reality(AR) component 128. In order to facilitate generating and providingauxiliary information to a healthcare professional that is relevant to acurrent aspect of a medical related procedure being performed by thehealthcare professional, the AR assistance module 110 can monitor andevaluate input received in real-time (e.g., during the performance ofthe procedure) regarding the healthcare professional's performance ofthe procedure. In one or more embodiments, this real-time input includesvideo or image data captured of the healthcare professional duringperformance of the healthcare related procedure (e.g., via camera 104 orone or more additional cameras in the healthcare professional'senvironment. For example, in some implementations, the healthcareprofessional can wear a device (e.g., user device 102) having a camera(e.g., camera 104) during performance of the procedure that can capturevideo and/or image data from a perspective of the healthcareprofessional during performance of the procedure. For example, deviceworn by the user can include an eyewear device (e.g., an AR eyeweardevice, a HUD, or another hands free device. According to this example,the camera can stream or otherwise provide the captured image and/orvideo data to the server device 108 in real-time (e.g., as it iscaptured). In other implementations, the video/image data can becaptured by one or more cameras located in the same room or environmentof the healthcare professional that are aimed at the healthcareprofessional during performance of the procedure. These cameras can alsostream or otherwise provide the captured image data to the AR assistancemodule 110 in real-time.

In addition to video/image feedback captured of the healthcareprofessional during performance of a healthcare related procedure, inother embodiments, the AR assistance module 110 can receive other formsof input that can be used to monitor and characterize various aspects ofa procedure being performed by the healthcare professional. In someimplementations, this additional feedback/input can include audiofeedback, motion feedback and/or biometric feedback capture performanceof the procedure by the healthcare professional. For example, the audiofeedback can include audio recorded during the procedure, such as speechof the healthcare professional, the patient, and/or potentially otherhealthcare professionals involved in the procedure. In another example,the audio feedback can include audio generated by a machine or othercomputing system associated with the procedure (e.g., an alarm signal, aheartbeat sound produced by a heart-rate monitor, etc.). The motionfeedback can include motion data generated from one or more motionsensors (e.g., accelerometers, gyroscopes, etc.) worn by the healthcareprofessional, the patient and/or other healthcare professionals involvedin the procedure. For example, the motion feedback can provideinformation regarding position, orientation and movement of a user,including specific positions, orientations and movements of respectivebody parts of the user. In another example, the motion feedback caninclude information regarding pressure or force applied to a patient bythe healthcare professional or medical instrument used by the healthcareprofessional (e.g., as captured via a motion sensor or pressure sensor).The motion feedback can also include motion data captured from a motionsensor associated with a medical device or instrument used during theprocedure. For example, this type of motion feedback can includeinformation regarding position, orientation and/or movement of themedical instrument relative to a patient's anatomy. The biometricfeedback can include biometric sensor feedback regarding physiologicalresponses of the healthcare professional, the patient and/or otherhealthcare professionals involved in the procedure. For example, thebiometric feedback can include information regarding a user's vitalsigns (e.g., body temperature, pulse rate, respiration rate, bloodpressure, etc.), stress levels, fatigue levels, attention levels, andthe like. Such biometric feedback can be captured via one or morebiometric sensors or biometric sensing devices worn by or otherwisephysically coupled to the healthcare professional, the patient, and/orother healthcare professionals involved in the procedure.

In addition to the various types of visual, audio, motion and biometricfeedback discussed above, in some implementations in which a medicalrelated procedure involves usage of a device configured to capture orgenerate data (in addition to the cameras and motion or biometricsensors discussed above), the AR assistance module 110 can also receivethis data as feedback. For example, in implementations in which themedical related procedure involves data entry by the healthcareprofessional into another computing system, the AR assistance module canalso receive feedback from the other computing system regarding the dataentry. For example, if the procedure involves entering patientinformation into a computing system, the AR assistance module 110 canreceive or access this patient information as it is entered. In anotherexample, if the procedure involves entering programming information intoa computing system that controls a medical device involved in theprocedure (e.g., an implantable medical device (IMD), a robotic device,an imaging device, a laser device, etc.), the AR assistance module 110can receive or access this programming information as it is entered. Inanother example, if the procedure involves entering billing information(e.g., billing codes, procedure codes, insurance information, etc.) intoa billing system, the AR assistance module 110 can receive this billinginformation as it is entered. In other implementations in which themedical related procedure involves usage of a medical device orinstrument configured to capture or generate data during a procedure,the AR assistance module 110 can also receive this data as input. Forexample, the medical device or instrument can be configured to captureand/or measure physiological data associated with the patient usingvarious sensors and provide this data to the AR assistance module. Inanother example, the medical device or instrument can include an imagingdevice used during a procedure that captures images of a patient's body(e.g., internal or external) during a medical procedure. For example,such imaging devices can include but are not limited to: radiographyimaging devices, magnetic resonance imaging devices, nuclear imagingdevices, ultrasound imaging devices, elastography imaging devices,tactile imaging devices, photoacoustic imaging devices, and the like. Inanother example, the AR assistance module 110 can receive image datacaptured from one or more cameras provided on or within medicalinstruments used during a procedure. For example, many invasive surgicaldevices such as probes, catheters, endoscopes, include imaging sensorsthat capture images of a patient's internal body during a surgicalprocedure to facilitate the procedure. Still in yet another example, theAR assistance module 110 can receive 2D and 3D imagining models of apatient's body or internal organs that are used and/or generated duringa surgical procedure to facilitate the procedure.

In one or more embodiments, the procedure identification component 112can be configured to analyze this visual, audio, motion, biometric, dataentry and/or medical device generated input as it is received inreal-time to identify and characterize various aspects of a procedurethat occur in real-time. This can include the type of procedure andvarious specific characteristics of the procedure, such as therespective actions and events of the procedure and characteristics ofthese actions or events. For example, in association with monitoring amedical procedure, the procedure identification component 112 canidentify the specific medical procedure being performed. The procedurecharacterization component 114 can further generate descriptiveinformation that describes the respective actions/steps performed by thehealthcare professional during the procedure based on the receivedinput. For example, in one or more embodiments, the procedurecharacterization component 114 can identify actions or steps beingperformed by the healthcare professional and characteristics of thoseactions or steps. The characteristics can vary depending on the type ofprocedure being performed and the type of input/feedback received. Invarious implementations, the characteristics described by the procedurecharacterization component 114 can generally describe what thehealthcare professional is doing, how the healthcare professional isdoing it, what the healthcare professional is using or applying and how(e.g., drug and dosage), the state of a patient, system or instrumentbefore performance of a procedural action or step, and the state of thepatient, system or instrument after performance of the procedural actionor step. For example, in some implementations, the characteristics canrelate to the physical position/orientation or movement of thehealthcare professional, a body part of the healthcare professional oran instrument operated by the healthcare professional, in associationwith performance of the procedural action/step. In another example, thecharacteristics can relate to a state, status, or appearance of systemsand equipment used in a procedure. In another example, thecharacteristics can include a physiological response of the healthcareprofessional, the patient, and/or other healthcare professionalsinvolved in the procedure. In some aspects, these physiologicalresponses can indicate mental states of the healthcare professionalthroughout the procedure (e.g., focused, confused, fatigued, stressed,concerned, etc.).

For example, with respect to an IV cannulation procedure, based onvisual, audio, motion, biometric, etc., input received during theprocedure, the procedure identification component 112 can identify ordetermine that an IV cannulation procedure is being performed. Theprocedure characterization component 114 can further generatedescriptive information regarding the steps/actions performed as theyare performed, such as but not limited to: preparation of the IV site,preparation of the tubing and the fluid bag, application of thetourniquet, selection of the catheter, insertion of the catheter,removal of the tourniquet, attaching the tubing, and opening of the IVline. The procedure characterization component 114 can further providedetailed information about the respective actions/steps observed, suchas for example: timing of the actions, order of the actions, instrumentsused, sizing of the instruments, location of the instruments, locationof the IV site, positions of the healthcare professional hands,physiological responses of the patient, physiological responses of thehealthcare professional (e.g., indicating stress, fatigue, etc.),pressure applied to the patient during insertion of the needle andtubing, dosage applied to the IV fluid, etc. It should be appreciatedthat the amount and type of the descriptive information will vary basedon the type of procedure being performed and the type of input/feedbackreceived by the AR assistance module during the procedure. Thus insummary, the procedure identification component 112 can be configured toanalyze the various input discussed above to identify a specificprocedure being performed and the procedure characterization component114 can further generate descriptive information that describes ornarrates the respective actions and events of the procedure as theyoccur.

In one or more embodiments, the AR assistance module 110 can beconfigured to monitor and assess a defined set of healthcare relatedprocedures. These procedures can include medical procedures, clericalprocedures and/or maintenance procedures. Each of the procedures can beassociated with a unique identifier or code that identifies the specificprocedure. The AR assistance module 110 can further access information(e.g., stored in memory 130, at one or more external information sources138, or another device/system), that identifies the respectiveprocedures the AR assistance module 110 is configured to monitor andevaluate. In some embodiments, prior to or during performance of ahealthcare related procedure, the procedure identification component 112can receive explicit information identifying the procedure beingperformed (e.g., the identifier for the procedure, the name of theprocedure, etc.). This can be because the procedure is scheduled in thesystem. For example, each medical bay in a facility may have proceduresdefined for it during the day. Or a nurse's schedule may have setprocedures for her to perform that day and in a specific order. And insome implementations, in association with beginning a medical relatedprocedure, the healthcare professional can enter (e.g., via a suitableinput mechanism) or otherwise provide, information to the user device102 identifying the specific procedure being performed. The user device102 can further provide this information to the AR assistance module110.

In other embodiments, the procedure identification component 112 candetermine the procedure being performed based on analysis of thereceived input data associated with performance of the procedure. Forexample, based on the received input (e.g., visual, audio, motion,biometric, data entry, medical device generated, etc.), the procedurecharacterization component 114 can generate one or more descriptiveparameters regarding characteristics of the procedure observed (e.g.,identified objects, people, words, actions or event, characteristics ofthose actions or events, etc.). The procedure identification component112 can further determine the type of procedure being performed bycomparing the one or more descriptive parameters with predefinedinformation (e.g., in memory 130, at one or more external informationsources 138, or at another device) that associates the one or moredescriptive parameters (e.g., the identified objects, people, words,actions or event, characteristics of those actions or events, etc.) orpatterns in the descriptive parameters with a defined type of procedure.

A data structure storing the procedure information may have thispredefined information as trigger circumstances. A procedure may havefour trigger circumstances like a certain image of a drug is captured inimage data, or a certain type of medical instrument is found in imagedata, or a certain phrase is captured in audio data (“I'm going to beadministering your prescription now”). Further, the data structurestoring the procedure information may specific how many of the triggercircumstances need to be meet for the system to detect the procedure isunderway. Thus, if the procedure identification component 112 detectsone or more trigger circumstances as predefined information, a proceduremay be detected on-the-fly.

For example, based on reception of image data including a healthcareprofessional preparing a patient's IV site, the procedurecharacterization component 114 can generate descriptive information thatindicates “IV site preparation” is performed or “an image of a patientsIV site” has been detected. The procedure identification component 112can further determine that the healthcare professional is performing anIV cannulation procedure based on data associating the determineddescriptive information with an IV cannulation procedure. In anotherexample, based on reception of audio input from healthcare professionalinvolved in a procedure stating “I'm preparing the IV site,” theprocedure characterization component 114 can determine and/or generatecorresponding descriptive data corresponding to keywords identified inthe audio signal (e.g., prepare, IV, and site). Based on these keyworddescriptors, trigger circumstances, and information associating thesekeyword descriptors with an IV cannulation procedure, the procedureidentification component 112 can determine the healthcare professionalis performing an IV cannulation procedure.

In some embodiments in which the procedure identification component 112does not receive direct or explicit information identifying a procedurebeing performed, in addition to the descriptive information generated bythe procedure characterization component 114, the procedureidentification component 112 can use contextual information tofacilitate identifying a procedure being performed. This contextualinformation can relate to but is not limited to: a location associatedwith the procedure (e.g., a specific room, area, sector, etc., of ahospital), a role of a user performing the procedure (e.g., a doctor, atype of the doctor, a nurse, etc.), a time of day, and/or patientinformation regarding a patient involved in the procedure. For example,certain locations in a healthcare environment can be associated with oneor more known procedures. Thus depending on where a user performing aprocedure is located, the procedure identification component 112 candetermine which process type is likely happening (e.g., operating room Bis often where cardiac procedures are performed, floor 3 is generallywhere rehabilitation procedures are performed, etc.). Further, ahealthcare professional's role or title can also limit the number andtype of procedures the person is authorized to perform. Thus theprocedure identification component 112 can limit detection of proceduresperformed by a certain healthcare professional to only those thehealthcare professional is authorized to perform. Still in otherembodiments, respective healthcare professionals can become associatedwith information indicating the procedures they are likely to perform(e.g., based on historical data regarding past performance of therespective healthcare professionals). The procedure identificationcomponent 112 can further limit detection of performance of a procedureby a particular healthcare professional based on those procedures thehealthcare professional is likely to perform. In another example,various procedures can be associated with performance at different timesof day or different days of the week. Thus in some implementations, theprocedure identification component 112 can restrict identification of acertain procedure being performed based on the time of day or day ofweek. Still on other implementation in which a procedure involves apatient, the procedure identification component 112 can restrictdetection of procedures associated with that patient based on thoseprocedures which the patient is scheduled to receive and/or authorizedto receive.

According to these embodiments, the AR assistance module 110 can includecontext component 118 to facilitate determining this contextualinformation, including but not limited to: a current location associatedwith the procedure, a role of a user performing the procedure, a time ofday, and/or patient information regarding a patient involved in theprocedure. For example, the context component 118 can determine anidentify of a location of healthcare professional performing aprocedure, a role of the healthcare professional, an identity of thehealthcare professional, an identity of the patient, and a time ofday/day of the week. The context component 118 can further accesspredefined information (e.g., stored in memory 130, at one or moreexternal information sources 138, or another device), that associatescertain procedures with one or more of: specific locations in ahealthcare facility, specific times of day/day of week, specifichealthcare professionals or healthcare professional roles, specificpatients, and the like. The context component 118 can employ variousknown techniques to determine a current location of a user. For example,these techniques can be based on global positioning system (GPS) dataassociated with the user or a user device 102 employed by the user,triangulation based locating methods, radio frequency identification tag(RFID) detection methods, monitoring user movement using cameras,beacons, and/or motion sensors, movement information monitored for theuser, and the like. The context component can also employ variousmechanisms to determine identities of healthcare professionals and/orpatients. For example, in some embodiments, the context component 118can determine an identity of a user based on image data captured of theuser and suitable facial recognition techniques. In other embodiments,the context component 118 can determine an identity of a user performinga procedure based in part on information associating the user with acertain device (e.g., user device 102) operated by the user and usage ofthat device by the user in association with performance of theprocedure.

The procedure characterization component 114 can further generatedescriptive information that describes various aspects of a procedureover the course of the procedure. In one or more embodiments, inaddition to facilitating identifying a procedure being performed, thisdescriptive information can be used by the procedure assessmentcomponent 116 to evaluate the performance of the procedure in real-timeover the course of the procedure. As noted above, the descriptiveparameters identified and/or generated by the procedure characterizationcomponent 114 can describe various aspects of a procedure beingperformed, including respective actions and events of the procedure andcharacteristics of those actions or events. For example, descriptiveinformation generated by the procedure characterization component overthe course of procedure can include but is not limited to: objectsassociated with a procedures or procedural environment, features of theobjects, people associated with a procedure or procedural environment,features of the people, procedural actions/steps performed,characteristics of the procedural actions or steps, physicalactions/motions performed by a user or an instrument, changes inappearance of a user or instrument, words spoken, sounds detected,physiological responses, and the like.

The mechanisms via which the procedure characterization component 114generates descriptive parameters for a procedural environment and/or aprocedure being performed by a healthcare professional can vary based onthe type of received input. For example, regarding video and/or stillimage data captured of a procedural environment and/or of healthcareprofessional performing a procedure, in one or more embodiments, theprocedure characterization component 114 can use various imagerecognition algorithms to identify objects in the image data (includingpeople/faces), and features of the objects (e.g., color, size, brand,relative location, orientation, etc.). In addition, by analyzing asequence or series of images (e.g., video data), the procedurecharacterization component 114 can identify actions, motions, behaviorsand facial expressions represented by the image data. Image recognitionalgorithms rely on matching, learning, or pattern recognition algorithmsusing appearance-based or feature-based techniques. Common techniquesinclude edges, gradients, histogram of oriented gradients (HOG), haarwavelets, and linear binary patterns. In various embodiments, theprocedure characterization component 114 can generate informationidentifying objects, people, facial expressions, actions, behaviors,motions, etc., appearing in image data using a variety of models,including but not limited to: extracted features and boosted learningalgorithms, bag-of-words models with features such as speeded-up robustfeatures (SURF) and maximally stable extremal regions (MSER),gradient-based and derivative-based matching approaches, Viola-Jonesalgorithm, template matching, and image segmentation and blob analysis.

Regarding audio feedback, the procedure characterization component 114can used pattern recognition techniques to identify uniquecharacteristics in an audio signal. Based on the unique characteristics,the procedure characterization component 114 can identify distinct wordsor sounds and generate descriptive information identifying the words orsounds. Regarding motion data received from motion sensors worn by thehealthcare professional, the patient, or another individual involved inthe procedure, the procedure characterization component 114 can identifypatterns in the motion data that correspond to known movements orgestures. For example, the procedure characterization component 114 candetermine if a user is moving a certain body part and how, if the useris walking, standing, sitting, etc. Likewise, based on motion sensordata regarding motion of tool or instrument, the procedurecharacterization component 114 can determine relative positions andorientations of the instrument (e.g., relative to the patient, theperson operating the instrument, etc.), and movements of the instrument.The procedure characterization component 114 can further generatedescriptive information defining the movements of a user and aninstrument throughout a procedure. The procedure characterizationcomponent 114 can also generate descriptive information that correspondsto input received from pressure sensors, biometric devices and othercomputing devices. For example, in response to reception ofphysiological information from a biometric device worn by a user (e.g.,the healthcare professional, the patient, other healthcareprofessionals, etc.) during a procedure, the descriptor generatorcomponent 114 can generate descriptive data that identifies thephysiological information (e.g., heart rate, vital signs, etc.). Inanother example, in response to entry of data into an associated computesystem regarding a drug dosage to apply to a patient, the procedurecharacterization component 114 can generate descriptive informationidentifying the event, the drug, and the specific dosage.

In various embodiments, the procedure characterization component 114 cangenerate descriptive information describing an aspect of a procedurebeing performed based on a combination of observed objects, persons,motions, physiological responses, and the like. For example, usingvarious machine learning techniques, patterns in various combinations ofsuch data can be learned which equate to defined procedural events. Inparticular, many procedural events can be characterized by a combinationof several descriptors, such as a sequence of motions involving acertain body part or instrument followed by a defined physiologicalresponse in the patient, and ending with position of the instrument orbody part at a defined position/location.

Machine learning is a type of artificial intelligence (AI) that providescomputers with the ability to learn without being explicitly programmedMachine learning focuses on the development of computer programs (e.g.,the procedure identification component 112, the procedurecharacterization component 114, and the procedure assessment component116) that can change when exposed to new data. Machine learningtechniques use that compiled data to detect patterns in the data andadjust program actions accordingly. Machine learning algorithms areoften categorized as being supervised or unsupervised. Supervisedalgorithms can apply what has been learned in the past to new data.Unsupervised algorithms can draw inferences from datasets. Deep learningis an aspect of AI that is concerned with emulating the learningapproach that human beings use to gain certain types of knowledge. Atits simplest, deep learning can be thought of as a way to automatepredictive analytics. While traditional machine learning algorithms arelinear, deep learning algorithms are stacked in a hierarchy ofincreasing complexity and abstraction. Each algorithm in the hierarchyapplies a non-linear transformation on its input and uses what it learnsto create a statistical model as output. Iterations continue until theoutput has reached an acceptable level of accuracy. The number ofprocessing layers through which data must pass is what inspired thelabel “deep.”

Thus in one or more embodiments, using one or more machine learning ordeep learning techniques evaluating feedback data (e.g., visual data,motion data, audible data, biometric data, medical device generateddata, etc.) received for many performances (e.g., by the same user ordifferent users) of the same procedure over time, patterns in thefeedback data can be identified which correlate to defined steps oractions of the procedure and characteristic of those steps and actions.This can be repeated for every procedure the AR assistance module 110may potentially evaluate. For each procedure, reference patterninformation can thus be developed that associates combinations and/orpatterns in image descriptors, audio descriptors, motion descriptors,physiological descriptors, and the like, with defined procedural eventsor actions and characteristics of the actions or events for theprocedure. The procedure characterization component 114 can access thisreference pattern information (e.g., stored in memory, at one or moreexternal information source, or at another device), to identify patternsin currently received feedback data (e.g., visual data, motion data,audible data, biometric data, medical device generated data, etc.)during performance of a procedure by a healthcare professional toidentifying corresponding reference patterns and associated proceduralactions and/or characteristics of those actions.

In one or more embodiments, the procedure assessment component 116 canbe configured to assess or evaluate a procedure being performed inreal-time based on the descriptive information generated/determined bythe procedure characterization component 114. In particular, the ARassistance module 110 can be configured to evaluate a set of knownmedical related procedures. Each of these procedures can have definedguidelines or protocols that define a desired manner for performing theprocedure. These guidelines/protocols can be defined by the healthcareorganization and/or government mandated. In some implementations, theseguidelines and protocols can be optimized over time using variousmachine learning techniques and data monitored for repeated or similarprocedures (as discussed infra with reference to optimization component604). Information identifying the respective procedures and theguidelines/protocols for the respective procedures can be included inmemory 130 (e.g., as procedure guideline/protocol information 132), atone or more external information sources 138, or another sourceaccessible to the AR assistance module. The guideline/protocolinformation for each procedure can include reference descriptiveparameters that describe various aspects of the procedure, including butnot limited to: required steps or actions for a procedure and timing forthe respective steps or actions, including an order of for performanceof the respective actions. The procedure guideline/protocol information132 can also include reference descriptive parameters that definecharacteristics of the required steps or actions, such as but notlimited to descriptive parameters related to: how the respectiveactions/steps should be performed, what supplies/instruments should beused, how to dispose of supplies, how to interact with a patient and/orother healthcare professionals during a procedure and the like. Inanother example, the guideline/protocol information can includedescriptive parameters regarding responsive actions/steps to be takenwhen various potential events, reactions, or complications arise duringa procedure.

In various embodiments, during performance of a procedure by ahealthcare professional, the procedure assessment component 116 can beconfigured to compare the descriptive parameters characterizing variousaspects of the procedure generated by the procedure characterizationcomponent 114 with the reference descriptive parameters included indefined guidelines/protocols for the procedure. Based on the comparison,the procedure assessment component 116 can determine what steps oractions included in the defined procedure guidelines/protocols thehealthcare professional is currently performing. The procedureassessment component 116 can further determine how the healthcareprofessional is performing various aspects of the procedure relative tothe defined guidelines/protocols for the procedure. In particular, inone or more embodiments, the procedure assessment component 116 candetermine if and when the healthcare professional is deviating from thedefined guidelines/protocols for the procedure or otherwise performingan aspect of the procedure incorrectly (e.g., against the definedguidelines and protocols for the procedure). The procedure assessmentcomponent 116 can also identify the specific aspect of the procedure thehealthcare professional is performing incorrectly.

For example, based on comparison of descriptive information generated bythe procedure characterization component 114 regarding steps/actions ofthe procedure that are performed and with reference descriptiveinformation for the procedure, the procedure assessment component 116can determine if the healthcare professional performed the step/actionout of order, if the healthcare professional is performing a step oraction too slow or too fast, if the healthcare professional skipped astep or action, if the healthcare performed an unnecessary step oraction, and the like. In another example, with respect to a specificaction or step of the procedure, the procedure assessment component 116can compare information generated by the procedure characterizationcomponent 114 describing one or more characteristic of the specificaction or step being performed by the healthcare professional withreference information defining required characteristics for theprocedural action. Based on the comparison, the procedure assessmentcomponent 116 can determine if the healthcare professional's performanceof the procedural action is on par with the required characteristics forthe procedural action. For example, with respect to a specific step of asurgical procedure, the procedure assessment component can determine ifthe healthcare professional is positioned correctly relative to thepatient, if the healthcare professional is operating on the correct sideand/or body part of the patient, if the healthcare professional is usingthe correct instruments, if the patient's vital signs are appropriatefor the step, if the healthcare professional's level of fatigue isacceptable, and the like. In addition to determining if and how ahealthcare professional has performed or is performing an aspect of aprocedure incorrectly, in various embodiments, the procedure assessmentcomponent 116 can also determine if and when a healthcare professionalhas performed or is performing an aspect of a procedure correctly.

In some embodiments, the procedure assessment component 116 can furthertailor its evaluation of a healthcare professional's performance of aprocedure based on a particular context of the procedure. For example,the manner in which certain medical procedures are performed are oftentailored to the patient based on the condition of the patient, thephysical health of the patient, health records of the patient, theseverity of the ailment of the patient, the sensitivity of the patient,the preferences of the patient, the age of the patient, the allergies ofthe patient, the mental stability of the patient, and the like. Suchconditional factors can affect the particular actions/steps of aprocedure that are performed, the manner in which they are performed,the pharmaceuticals that are applied, the dosage of the pharmaceuticalsand the like. In some implementations, prior to performing a procedure,a clinician can determine and provide information regarding variousaspects of a procedure that have been tailored to a particular patient.This information can be made accessible to the procedure assessmentcomponent (e.g., at one or more external information sources 138 oranother device). In another example, the manner in which certain medicalprocedures are performed can depend on the equipment and supplies thatare available. Thus in various embodiments, the context component 118can determine such contextual information regarding a patient associatedwith a procedure, the equipment/supplies available, and the environmentof the procedure (e.g., by identifying the relevant contextualinformation in one or more external information sources 138). Theprocedure assessment component 116 can further tailor evaluation of theprocedure based on such contextual information.

In some embodiments, in addition to generating descriptive informationthat narrates a procedure being performed, the procedurecharacterization component 114 can further evaluate an environment inwhich the procedure is occurring or will occur to characterize theenvironment. For example, using image data captured from theenvironment, the procedure characterization component 114 can identifythe objects in the environment, as well as the state or status of theobjects, including medical instruments, supplies and devices (e.g.,on/off, used/unused, working properly/improperly, etc.,), and therelative positions of the objects. The procedure characterizationcomponent 114 can also access information (e.g., stored in memory 130,at an external information sources 138, or another device), thatprovides information regarding the state or status of the objects. Basedon information describing the procedure environment, the procedureassessment component 116 can make determinations regarding whether theenvironment is properly prepared and equip for performance of theprocedure in accordance with predetermined quality control standards.The procedure assessment component 116 can further assess the proceduralenvironment at various points throughout the procedure and uponcompletion of the procedure to determine whether all instruments,supplies, equipment and personnel that should be present are present,whether the instruments, supplies, equipment and personnel are in thecorrect positions, and whether the instruments, supplies, equipment andpersonnel are in the correct state/status.

According to these embodiments, the procedure guideline/protocolinformation 132 can also include reference information identifyingrequirements for the procedural environment at various points throughoutthe procedure, such as but not limited to: what equipment and suppliesshould be included in the environment, status/state of theequipment/supplies at various points throughout the procedure (e.g.,on/off, operating properly/improperly, used/unused, etc.), relativepositions of the equipment and supplies at various points throughout theprocedure (e.g., external to patient, in patient, in sanitation bag, intrash unit, etc., what people that should be located in the proceduralenvironment, where and/when, how those people should appear (e.g., whatthey should be wearing), etc. Based on information determined by theprocedure characterization component regarding the proceduralenvironment, the procedure assessment component 116 can further assessthe procedural environment prior to beginning the procedure, at variouspoints throughout the procedure, and upon completion of the procedure,to determine whether the procedural environment is on par with therequired guidelines. For example, the procedure assessment component 116can determine if certain supplies or equipment is missing from theprocedure environment, if certain supplies or equipment is functioningimproperly, if supplies/tools were unused when they should have been,whether certain supplies/tools that were used were not disposedcorrectly (e.g., a sponge was left in the patient), and the like. Inanother example, the procedure assessment component 116 can determine ifa healthcare professional gets the wrong supplies/equipment out for thesurgery.

The feedback component 120 can be configured to determine and/orgenerate feedback information in real-time in response to determinationsmade by the procedure assessment component 116 regarding performance ofa procedure by a healthcare professional and/or the proceduralenvironment. The feedback component 120 can further provide thisfeedback to a healthcare professional that is performing a procedure, auser supervising performance of the procedure, or another entity (e.g.,a remote monitoring system/service, an administrative personnel, etc.).Based on received during performance of a procedure feedback regardingan aspect of a procedure a healthcare profession has performed or iscurrently performing incorrectly, the healthcare professional can takeappropriate action to correct the mistake immediately, therebyminimizing potential complications of the mistake down the road. Thishas a strong benefit to patients, payors, practitioners, clinicians,governments, and healthcare systems in general. Any reduction inmistakes can save lives and reduce cost.

In one or more embodiments, the feedback component 120 can includecorrection component 122 to determine and/or generate correctivefeedback regarding an aspect of a procedure that has been or is beingperformed incorrectly and/or an aspect of a procedure environment thatis improper. For example, based on a determination by the procedureassessment component 116 that a healthcare professional has skipped aprocedural step or action, the correction component 122 can determine orgenerate information that identifies the skipped step or action andinforms the healthcare professional regarding the skipped step oraction. In another example, based on a determination by the procedureassessment component 116 that a healthcare professional is performing anunnecessary step or action, the correction component 122 can determineor generate information that identifies the unnecessary step or actionand informs the healthcare professional regarding the unnecessary stepor action. In another example, based on a determination by the procedureassessment component 116 that a healthcare professional is performing astep or action out of order, the correction component 122 can determineor generate information that identifies the step or action performed inthe incorrect order. The information can further describe the correctorder for the step. For example, the information can identify one ormore steps or actions that should be performed prior to the step oraction the healthcare professional is initiating or currentlyperforming.

In another example, based on a determination by the procedure assessmentcomponent 116 that a healthcare professional has performed or isperforming a step or action incorrectly, the correction component 122can determine or generate information that identifies the step oraction. In some embodiments, correction component 122 can furtherdetermine or generate information regarding the specific aspect of thestep or action that is incorrect. The correction component 122 can alsodetermine or generate information regarding how to correct the error.For example, the correction component 122 can retrieve the referenceinformation associated with the particular step or action in a referencedata structure that defines how the step or action, or the specificaspect of the step or action, should be performed. The data structurefor the procedure may have text, video, images, and audio stored withrespect to each step or action in a procedure. These all may be used inassisting the healthcare professional through an AR interface.

Regarding an identified error with respect to the proceduralenvironment, the correction component 122 can determine and/or generateinformation that identifies the aspect of the procedure environment thatdoes not comply with the defined guidelines/requirements for theprocedure. For example, based on a determination that a particularinstrument needed for a procedure is missing from the proceduralenvironment, the correction component 122 can determine and/or generateinformation indicating the missing instrument. In another example, basedon a determination that a sponge used during a surgical procedure hasnot been removed from the patient upon initiation of closing up thepatient's surgical site, the correction component 122 can determineand/or generate information indicating the incorrect position and/ordisposal of the sponge. In another example, based on a determinationthat an assisting nurse is located near a patient's feet at a timeduring a procedure when she is needed to be located near the patient'sheat to assist a primary healthcare professional with a step of aprocedure, the correction component 122 can determine and/or generateinformation indicating the incorrect position of the nurse andidentifying the correct position. In another example, based on adetermination that a healthcare professional gets the wrongsupplies/equipment out for the surgery, the correction component cangenerate corrective feedback indicating the wrong supplies/equipmenthave been selected and provide the healthcare professional with feedbackindicating the wrong supplies/equipment and instructing the healthcareprofessional get the right supplies/equipment out so as not to open andwaste supplies/equipment that aren't needed.

The feedback component 120 can also include praise component 124 todetermine and/or generate praise information regarding one or moreaspects of a procedure a healthcare professional is performing or hasperformed correctly. For example, over the course of a procedure, if theprocedure assessment component 116 determines that the healthcareprofessional performed an action or step of the procedure correctly, thepraise component 124 can generate information identifying the step oraction that was performed correctly.

The feedback component 120 can also include guidance component 126 tofacilitate providing a healthcare professional with guidance informationthat guide the healthcare professional throughout a procedure. Forexample, this guidance information can inform a healthcare professionalregarding how to perform each step of a procedure and when. Suchinformation can be useful to train healthcare professionals how toperform new procedures and/or to guide a healthcare professional thatneeds or desires such step by step assistance. For example, in someembodiments, the guidance component 126 can retrieve referenceguideline/protocol information for a procedure that is being performed.The reference guideline/protocol information can defines how to performthe procedure, including how to perform each step or action of theprocedure, when to perform each step or action, and the characteristicsof the respective steps or actions. The guidance component 126 canfurther provide the healthcare professional with information for eachstep at the time when the step is to be performed, generating a livestep by step tutorial explaining how to perform each step of theprocedure. In some embodiments, the guidance component 126 can determinewhat information to provide the healthcare professional at variouspoints based on information determined by the procedure assessmentcomponent 116 regarding when certain steps or actions are completed orinitiated by the healthcare professional. For example, with respect to aprocedure including 10 steps, based on information determined by theprocedure assessment component 116 that the healthcare professional hascompleted step 1 and/or is initiating step 2, the guidance component 126can determine or retrieve information describing how to perform step 2and provide this information to the healthcare professional at thistime.

Further in some embodiments, the feedback component 120 can access pastperformance information (e.g., stored in memory 130, at one or moreexternal information sources 138, or another device) recorded for thehealthcare professional regarding the healthcare professional's pastperformance of the same procedure that is currently being performed bythe healthcare professional (e.g., via recording component 602 discussedinfra with reference to FIG. 6). For example, each time the healthcareprofessional's performance of a particular procedure is evaluated by theAR assistance module 110, the AR assistance module (e.g., via recordingcomponent 602) can generate report information that documents theinformation determined and/or generated by the procedure identificationcomponent 112, the procedure assessment component 116 and/or thefeedback component 120 regarding the healthcare professionalsperformance, such as information regarding what steps/actions wereperformed correctly and incorrectly, what aspects of those actions wereperformed correctly or incorrectly, duration of performance of therespective steps, and the like. The feedback component 120 can furthercompare past performance information with current performanceinformation to determine, in real-time, how the healthcare professionalis performing respective steps of a procedure relative to pastperformance of the same steps. The feedback component 120 can furtherdetermine and provide the healthcare professional with information thatindicates how a healthcare professional's current performance of aprocedural step compares with her past performance of the same step. Forexample, with respect to a surgical procedure that involves 10procedural steps, the feedback component 120 can access past performanceinformation indicating the success rate for performance of each step.For example, such information may indicate that the healthcareprofessional performs steps 1-5 correctly 90% of the time, yet performsstep 6 incorrectly 50% of the time. Thus when the healthcareprofessional reaches step 6 during a current performance of theprocedure, if the procedure assessment component 116 determines that thehealthcare professional performs step 6 incorrectly, the correctioncomponent 122 can determine and/or generate information regarding theincorrect performance of step 6. The correction component 122 can alsogenerate feedback information indicating the 50% success rate of step 6in the past, and the degree to which her success rate for step 6 hasdecreased based on the current error. Alternatively, if the procedureassessment component 116 determines that the healthcare professionalperforms step 6 correctly, the praise component 124 can generate andprovide information to the healthcare professional regarding the correctperformance and the improvement to her success rate for step 6.

Further, in some embodiments, the guidance component 126 can beconfigured to automatically generate and provide guidance informationregarding how to perform certain steps of a procedure based on pastperformance information for those steps. For example, based on pastperformance information indicating that this healthcare professionalregularly performs step 6 incorrectly, when the healthcare professionalreaches step 6 during the procedure, the guidance component 126 cangenerate/provide the healthcare professional with guideline informationdescribing how to perform step 6 correctly.

The manner in which the feedback component 120 provides feedback to ahealthcare professional (or other individual/entity) in association withperformance of a healthcare procedure can vary. However, in variousexemplary embodiments, the AR assistance module 110 can include ARcomponent 128 to facilitate providing feedback determined/generated bythe feedback component 120 to a healthcare professional using ARvisioning techniques. According to these embodiments, the AR component128 can be configured to generate overlay data comprising or otherwiserepresenting the feedback information determined/generated by thefeedback component 120. For example, the overlay data can include text,an image, a video, a symbol, a marker, an icon, a hyperlink or othersuitable visual data object that can be projected onto a display thatviews or includes a visualization of the healthcare professional'senvironment from the perspective of the healthcare professional. Forexample, in various embodiments, the user device 102 includes a ARdevice that is worn by the healthcare professional during performance ofthe procedure. In some embodiments, the display 106 of the user device102 can include a transparent display through which the healthcareprofessional views the environment. In another example, the display 106can present a live video or image of the healthcare professional'senvironment. Regardless of the type of display, the overlay data can beconfigured for rendering on the display over or within a view of theenvironment.

In some embodiments, the overlay data can be spatially aligned with oneor more objects included in the environment and viewed on or through thedisplay. For example, if a healthcare professional prepares an incisionsite incorrectly, the overlay data can be rendered spatially on or nearthe incision site. According to these embodiments, the AR component 128can determine the object to spatially align the overlay data with basedon the feedback information and/or the specific aspect of the procedurefor which the overlay data is based. The AR component 128 can furtherprovide the user device 102 (or the device at which the overlay data isto be displayed), with the overlay information and informationidentifying the object to spatially align the overlay data with. Theuser device 102 can further be configured to render the overlay datarelative to the object included in the display accordingly.

For example, in some embodiments, the feedback information generated bythe feedback component 120 can be determined or selected from a set ofdefined feedback options for each procedure. According to theseembodiments, each feedback option can be associated with information(e.g., included in memory 130, at one or more external informationsources 138, or at another) identifying an object or objects forassociation with overlay information. The AR component 128 can thusdetermine the object to spatially align the overlay data with based onpreviously determined information associating the feedback informationwith the object. In another embodiment, the AR component 128 canidentify key words or terms associated with the feedback informationthat the overlay data includes or represents. For example, the feedbackinformation can indicate that the “incision site is incorrectlyprepared.” With theses implementations, based on recognition of the keyword “incision site,” the AR component 128 can determine the object toassociate the overlay data with is the incision site. In someimplementations of this embodiment, the AR component 128 can also employindex information (e.g., stored in memory, at one or more externalinformation sources 138, or another device), that associates previouslydetermined key terms that may potentially be included in feedbackinformation determined or generated by the feedback component 120 withobjects. In some embodiments, each procedure can be associated with anindex tailored to that procedure that includes key terms and associatedobjects relevant to that procedure. In some implementations, the indexcan further include hierarchy information that indicates what key termand/or associated object to select when two or more key terms areincluded in feedback information to be represented by the overlay data.For example, if feedback information indicates that “the location of thephysician's instrument relative to the incision site is incorrect,” theAR component 128 can recognize both the terms “instrument” and “incisionsite.” The hierarchy information however can indicate the term“instrument” receives priority over the term “incision site” and the ARcomponent 128 can select the incision site over the instrument as theobject for associating the overlay data.

In some embodiments in which the overlay data can be interacted with.For example, in embodiments in which the overlay data includes a video,the video can be selected for playback via the display 106. Forinstance, the video can include a visual demonstration regarding how toperform a particular aspect of a procedure. According to this example,the healthcare professional can stop his or her performance of aprocedure to watch the provided video prior to continuing with theprocedure. In another example, in which the overlay data includes ahyperlink, the healthcare professional can select the hyperlink toretrieve the information associated with the hyperlink.

FIGS. 2-4 present example AR visualizations including auxiliaryinformation presented to a clinician regarding current performance of ahealthcare procedure by the clinician in accordance with one or moreembodiments described herein. One or more aspects of the example ARvisualizations demonstrate the features and functionalities of system100 (and additional systems described herein).

With reference to FIGS. 1 and 2, visualization 200 depicts a clinician202 performing a medical procedure on a patient 204. In order tofacilitate performance of the medical procedure by the clinician 202,the clinician is wearing an AR device 206 that includes a transparentdisplay (not shown) through which the clinician 202 can view hisenvironment clearly (e.g., display 106). In various embodiments, the ARdevice 206 can be or include user device 102. Further, the AR device 206can include or be communicatively coupled to the AR assistance module110 to facilitate monitoring the performance of the procedure by theclinician 202 in real-time and providing the clinician 202 with overlaydata that guides or assists the clinician 202 with performing theprocedure.

In the embodiment shown, the clinician 202 is currently performing anaction of the medical procedure that includes entering IV informationinto a computing system 208 that controls concentration of IV fluidsupplied to the patient 204. Based on feedback received by the ARassistance module 110 and patient information indicating the proper IVconcentration for the patient, the AR assistance module 110 hasdetermined that the clinician 202 has entered the wrong IV concentrationfor the patient 204. In one or more embodiments, the feedback caninclude image data captured of the clinician entering the incorrect IVconcentration. For example, the image data can include (e.g., videoand/or still images) captured via a camera on the AR device 206 (notshown) with a field of view that corresponds or substantiallycorresponds to that of the clinician (e.g., with camera 104). In anotherimplementation, the image data can be captured via another camera in theoperating room. In other embodiments, the feedback can include theactual concentration information data entered into the computing system208 by the clinician.

As a result of the detected error, the AR assistance module 110 hasgenerated overlay data including feedback regarding the wrong IVconcentration. The overlay data is further rendered on the display (notshown) of the AR device 206 so that it is seen by the clinician overlaidonto the current view of the procedural environment viewed through thedisplay. For example, in accordance with visualization 200, this currentview includes at least the touch screen of the computing system 208 viawhich the clinician is entering information and the IV bag. The overlaydata includes a semi-transparent pop-up display window 210 with text anda warning symbol indicating the error of entering the wrong IVconcentration. The overlay data also includes a line connecting thedisplay window 210 to the actual IV bag, which is highlighted withadditional coloration data 212 that draws attention to the IV bag. In anaspect, the coloration data 212 can be a bright color (e.g., red,yellow) or flash such that the clinician's attention is drawn to the IVbag. In one or more implementations, the overlay data can remain in theclinician's view until the error is corrected, at which time it isdismissed. For example, based on a determination by the AR assistancemodule 110 that the clinician has entered the correct IV concentrationinformation, the AR assistance module 110 can provide a message to theuser device 102 instructing the user device to remove the overlay datafrom the display.

It should be appreciated that the appearance and location of the overlaydata (e.g., the display window 210 and the coloration data 212) ismerely exemplary and intended to convey the concept of what is actuallyviewed by the clinician 202 through the AR device 206. However, theappearance and location of the overlay data in visualization 200 is nottechnically accurate, as the actual location of the overlay data wouldbe on the glass/display of the AR device 206 which is not visible giventhe position of the clinician in the image.

With reference to FIGS. 1 and 3, visualization 300 depicts anotherclinician 302 performing a medical procedure on a patient 304. In orderto facilitate performance of the medical procedure by the clinician 202,the clinician is wearing an AR device 306 that includes a transparentdisplay (not shown) through which the clinician 302 can view herenvironment clearly (e.g., display 106). In various embodiments, the ARdevice 306 can be or include user device 102. Further, the AR device 306can include or be communicatively coupled to the AR assistance module110 to facilitate monitoring the performance of the procedure by theclinician 302 in real-time and providing the clinician 302 with overlaydata that guides or assists the clinician 302 with performing theprocedure.

In the embodiment shown, the clinician 302 is currently operating on thepatient 304. Based on feedback received by the AR assistance module 110and information regarding the medical supplies that should be accessibleto the clinician 302 at this stage of the procedure, the AR assistancemodule 110 has determined that a 12 gauge needle is missing from thesupply table. In one or more embodiments, the feedback can include imagedata captured of the procedural environment via a camera on the ARdevice 306 (not shown) with a field of view that corresponds orsubstantially corresponds to that of the clinician (e.g., with camera104). In another implementation, the image data can be captured viaanother camera in the operating room. As a result of the detected error,the AR assistance module 110 has generated overlay data includingfeedback regarding the missing 12 gauge needle. The overlay data isfurther rendered on the display (not shown) of the AR device 306 so thatit is seen by the clinician 302 overlaid onto the current view of theprocedural environment viewed through the display. For example, inaccordance with visualization 300, this current view includes at leastthe operating area of the patient (e.g., the patient's stomach), aportion of the assisting clinician and the supply table. The overlaydata includes a semi-transparent pop-up display window 308 with text anda warning symbol indicating missing 12 gauge supply needle. The overlaydata also includes a line connecting the display window 308 to agraphical image 310 of a 12 gauge supply needle that is overlaid ontothe table to draw the clinician's attention to the supply table and themissing needle.

It should be appreciated that the appearance and location of the overlaydata (e.g., the display window 308 and the graphical image 310) ismerely exemplary and intended to convey the concept of what is actuallyviewed by the clinician 302 through the AR device 306. However, theappearance and location of the overlay data in visualization 300 is nottechnically accurate, as the actual location of the overlay data wouldbe on the glass/display of the AR device 306 which is not visible giventhe position of the clinician in the image.

With reference to FIGS. 1 and 4, visualization 400 depicts anotherclinician 402 performing a medical procedure on a patient 404. In orderto facilitate performance of the medical procedure by the clinician 402,the clinician is wearing an AR device 406 that includes a transparentdisplay through which the clinician 402 can view his environment clearly(e.g., display 106). In various embodiments, the AR device 406 can be orinclude user device 102. Further, the AR device 406 can include or becommunicatively coupled to the AR assistance module 110 to facilitatemonitoring the performance of the procedure by the clinician 402 inreal-time and providing the clinician 302 with overlay data that guidesor assists the clinician 402 with performing the procedure.

In the embodiment shown, the clinician 402 is currently operating on thepatient 404. Based on feedback received by the AR assistance module 110and information regarding the defined steps of the medical procedure,the AR assistance module 110 has determined that the clinician isinserting the catheter instrument without having properly prepared theincision site. In one or more embodiments, the feedback can includeimage data captured of the procedural environment via a camera on the ARdevice 406 (not shown) with a field of view that corresponds orsubstantially corresponds to that of the clinician (e.g., with camera104). In another implementation, the image data can be captured viaanother camera in the operating room. In another implementation, thefeedback can be based on motion data captured via one or more motionsensors worn by the clinician 402 and/or attached to the catheterinstrument. As a result of the detected error, the AR assistance module110 has generated overlay data including feedback regarding the failureto prepare the incision site before performing the insertion step. Theoverlay data is further rendered on the display (not shown) of the ARdevice 406 so that it is seen by the clinician 402 overlaid onto thecurrent view of the procedural environment viewed through the display.For example, in accordance with visualization 400, this current viewincludes at least the operating area of the patient 404. The overlaydata includes a semi-transparent pop-up display window 408 with text anda warning symbol instructing the clinician to stop and prepare theincision site before insertion.

Again, it should be appreciated that the appearance and location of theoverlay data (e.g., the display window 410) is merely exemplary andintended to convey the concept of what is actually viewed by theclinician 402 through the AR device 406. However, the appearance andlocation of the overlay data in visualization 400 is not technicallyaccurate, as the actual location of the overlay data would be on theglass/display of the AR device 406.

FIG. 5 provides a flow diagram of an example method 500 for providingauxiliary information regarding healthcare procedure performance inreal-time using AR in accordance with one or more embodiments describedherein. In various embodiments, method 500 can be performed by system100 and the like using user device 102, server device 108 and/or one ormore external information sources 138. In one implementation of method500, a user (i.e., a healthcare professional) working in a healthcareenvironment wears an AR device (e.g., user device 102) throughout theworkday. For example, the AR device can include an eyewear device (e.g.,a HUD, glasses, goggles, or the like) that allows the user to see theenvironment through a transparent display (e.g., display 106). Theeyewear device can also include a camera (e.g., camera 104), thatcaptures video and/or still images of the user's environment from theperspective of the user. In other implementations, one or more camerascan be positioned at various locations throughout the user's environmentand capture image data (e.g., video and/or still images) of the user asthe user goes about his or her workday.

With reference to FIGS. 1 and 5, at 502, a camera captures a currentview of the user. For example, the camera can capture a video of theuser's environment from the user's perspective. In accordance withsystem 100, the video data can be provided to AR assistance module 110in real-time (e.g., streamed as it is captured) for processing by the ARassistance module 110. At 504, the video data is analyzed to determinewhether it appears the user is performing a procedure or startingperformance of a procedure (e.g., via procedure identification component112). If the procedure identification component 112 determines that aprocedure is not being started, the AR assistance module 110 cancontinue to analyze new video data as it is received until it determinesthe user is starting a procedure. At 506, based on a determination thatthe user has started a procedure, the AR assistance module determineswhether guidelines are set for the procedure (e.g., via procedureassessment component 116).

In some embodiments, if guidelines are not set for the procedure, at 508the AR assistance module 110 (e.g., via the procedure characterizationcomponent 114, the procedure assessment component 116, and/or therecording component 602), can record/monitor the procedure beingperformed to facilitate developing future guidelines for the procedure.For example, procedure characterization component 114 and/or theprocedure assessment component 116 can determine descriptive informationfor the procedure regarding the various actions or steps performed andthe characteristics of those actions or steps. The descriptiveinformation generated for the procedure can be stored (e.g., in memory130, at one or more external information sources 138, or another device)to facilitate evaluating the procedure to develop guidelines for theprocedure. For example, the descriptive information can be combined withadditional descriptive data regarding performance of the same procedure(by the user and/or other users). The combined data can be processedusing one or more machine learning or deep learning techniques todevelop guidelines regarding optimal steps for the procedure andcharacteristics of the steps.

If guidelines are set for the procedure, at 510, the AR assistancemodule 110 can compare (e.g., using the procedure assessment component116) the user actions (and characteristics of the actions, as determinedvia the procedure characterization component 114) with the guidelines.Based on the comparison, at 512, the AR assistance module 110 candetermine whether the user deviates from the guidelines. If the userdoes not deviate the guidelines, the AR assistance module can continuesto monitor the user's performance In some implementations, if the ARassistance module can also provide the user with praise informationregarding the good performance. If however the user does deviate fromthe guidelines, at 516, the AR assistance module 110 can determinefeedback for providing to the user based on the deviation, and at 518feedback can be provided to the user in real-time using AR (e.g., atuser device 102). In other implementations, if the user does deviatefrom the guidelines, at 514, the AR assistance module can notify theuser's supervisor in real-time. Details regarding generating and sendingsuch notifications are described infra with reference to FIG. 7 andnotification component 702.

FIG. 6 illustrates a block diagram of an example system 600 thatfacilitates providing auxiliary information regarding healthcareprocedure performance using AR in accordance with one or moreembodiments described herein. System 600 includes same or similarfeatures and functionality as system 100 with the addition of recordingcomponent 602 and optimization component 604 to the AR assistance module110. Repetitive description of same or similar functionalities of likecomponents described in prior embodiments is omitted for sake ofbrevity.

In one or more embodiments, the AR assistance module 110 can includerecording component 602 to record and compile data received andprocessed by the AR assistance module 110 over time. This compiled datacan be regularly processed by the optimization component 604 using oneor more machine learning techniques and/or deep learning techniques tofacilitate automatically optimizing or enhancing the accuracy andquality of determinations made by the procedure identification component112, the procedure characterization component 114, and the procedureassessment component 116.

For example, each time the AR assistance module is used to monitor andevaluate a procedure performed by a healthcare employee, the recordingcomponent 602 can record or store (e.g., in memory 130, in one or moreexternal information sources 138, or another device) informationregarding the received input used to evaluate the procedure. Thisinformation can include for example, visual (e.g., image and/or video)data received, audio data received, motion data received, biometric datareceived, machine generated data, and the like. The recording component602 can further record descriptive information determined by theprocedure characterization component 114 based on the received input.This information can include for example, information regarding objectsand people detected, information regarding movement of the people andobjects, information identifying the procedure being performed,information regarding actions or steps performed (e.g., as determinedbased on the people, objects and movements, and other inputs),information regarding characteristics of the procedure actions/steps,and the like. The recording component 602 can record such informationfor all procedures monitored by the AR assistance module over timeperformed by various healthcare professionals. Using this compiled dataas well as data available at one or more external information sources138, the optimization component 604 can perform one or more machinelearning and/or deep learning techniques to improve the determinationsmade by the procedure identification component 112 regarding accuratelyidentifying a procedure that is being performed. In addition, using thiscompiled data as well as data available at one or more externalinformation sources 138, the optimization component 604 can perform oneor more machine learning and/or deep learning techniques to improve thedeterminations made by the procedure characterization component 114regarding accurately characterizing the various actions andcharacteristics of a procedure being performed and the proceduralenvironment.

Similarly, for each procedure evaluated, the recording component 602 canrecord or store information (e.g., in memory 130, in one or moreexternal information sources 138, or another device) or regarding thevarious assessments made by the procedure assessment component 116 basedon the data determined and/or generated by the procedurecharacterization identification component 114 and the procedurecharacterization component. For example, the recording component 602 canrecord information regarding whether and how the user's performance ofthe procedure deviates from the defined guidelines/protocols for theprocedure. The recording component 602 can also record informationregarding whether and how and aspect of the procedural environment is inaccordance or discord with the defined guidelines/protocols forprocedure. The recording component 602 can record such information forall procedures monitored by the AR assistance module 110 over timeperformed by various healthcare professionals. Using this compiled dataas well as data available at one or more external information sources138, the optimization component 604 can perform one or more machinelearning and/or deep learning techniques to improve the determinationsmade by the procedure assessment component 116 regarding accuratelydetermining whether and how a user's performance of a procedure and/orprocedural environment deviates from guidelines/protocols for theprocedure that is being performed. In some embodiments, based on thecompiled data and information regarding patient and financial outcomesassociated with a monitored procedure, using machine learning and/ordeep learning techniques, the optimization component 604 can modify thedefined guidelines/protocols for respective procedures.

In order to provide for or aid in the numerous inferences describedherein, optimization component 604 can examine the entirety or a subsetof the data to which it is granted access and can provide for reasoningabout or infer states of the system (e.g., system 600 and the like),environment, etc. from a set of observations as captured via eventsand/or data. An inference can be employed to identify a specific contextor action, or can generate a probability distribution over states, forexample. The inference can be probabilistic (e.g., the computation of aprobability distribution over states of interest can be based on aconsideration of data and events). An inference can also refer totechniques employed for composing higher-level events from a set ofevents and/or data.

Such an inference can result in the construction of new events oractions from a set of observed events and/or stored event data, whetheror not the events are correlated in close temporal proximity, andwhether the events and data come from one or several event and datasources. Various classification (explicitly and/or implicitly trained)schemes and/or systems (e.g., support vector machines, neural networks,expert systems, Bayesian belief networks, fuzzy logic, data fusionengines, etc.) can be employed in connection with performing automaticand/or inferred action in connection with the claimed subject matter.

A classifier can map an input attribute vector, x=(x1, x2, x4, x4, xn),to a confidence that the input belongs to a class, such as byf(x)=confidence(class). Such classification can employ a probabilisticand/or statistical-based analysis (e.g., factoring into the analysisutilities and costs) to prognose or infer an action that a user desiresto be automatically performed. A support vector machine (SVM) is anexample of a classifier that can be employed. The SVM operates byfinding a hyper-surface in the space of possible inputs, where thehyper-surface attempts to split the triggering criteria from thenon-triggering events. Intuitively, this makes the classificationcorrect for testing data that is near, but not identical to trainingdata. Other directed and undirected model classification approachesinclude, e.g., naïve Bayes, Bayesian networks, decision trees, neuralnetworks, fuzzy logic models, and probabilistic classification modelsproviding different patterns of independence can be employed.Classification as used herein also is inclusive of statisticalregression that is utilized to develop models of priority.

FIG. 7 illustrates a block diagram of another example system thatfacilitates providing auxiliary information regarding healthcareprocedure performance using AR in accordance with one or moreembodiments described herein. System 700 includes same or similarfeatures and functionality as system 600 with the addition ofnotification component 702, audio feedback component 704 and live remoteassistance component 706 to the AR assistance module 110. Repetitivedescription of same or similar functionalities of like componentsdescribed in prior embodiments is omitted for sake of brevity.

In various embodiments, in addition to providing a user with feedbackregarding performance of a procedure or a procedural environment usingAR visualizations, the AR assistance module 110 can also be configuredto generate and provide notifications, audio feedback and/or live remoteassistance during the procedure. For example, the notification component702 can be configured to generate notifications in real-time regardingan aspect of a procedure being performed that deviates from the definedguidelines/protocols for the procedure. Such notifications can beprovided to an entity other than the user performing the procedure. Forexample, based on a determination (e.g., by procedure assessmentcomponent 116) that aspect of a procedure being performed by ahealthcare professional deviates from the defined guidelines/protocolsfor the procedure, the notification component 702 can be configured togenerate and send a notification regarding the deviation to a supervisorof the healthcare professional or another suitable entity. For example,the notification can be sent to a device of the supervisor in the forman electronic message, a phone call, or the like.

In some embodiments, the entity (and associated device) that thenotification component 702 sends the notification to can be based on thetype of the notification or the reason for the notification. Accordingto these embodiments, the notification component 702 can determine atype of the notification based on the deviation for which thenotification is based. For example, if the notification concerns missingor inoperative supplies or equipment, the notification component 702 canbe configured to send the notification to a particular maintenanceemployee. In another implementation, certain potential errors can beassociated with different severity classifications. According to thisimplementation, based on a determination that a particular aspect of aprocedure deviates from protocol, the notification component 702 canaccess information (e.g., stored in memory 130, at one or more externalinformation sources 138, or another device), that indicates aclassification for the current error. For example, if the error isperformance of step 2 of a particular procedure incorrectly, thenotification component 702 can access information that indicates aseverity classification for performance of step 2 of this particularprocedure incorrectly. Based on the severity classification, thenotification component 702 can be configured to send the notification toa defined entity that is instructed to supervise notifications havingthat severity classification. In another aspect, the notificationcomponent 702 can include information indicating the severity of theerror in the notification. The type of the notification can also varybased on the severity classification (e.g., a code blue notification canindicate moderate severity, while a code red notification indicates highseverity).

In some implementations, the notification component 702 can beconfigured to send notifications for only certain types of errors havinga severity classification above a defined threshold level. In yetanother implementation, the notification component 702 can be configuredto monitor the number, frequency and/or severity levels associated witherrors or deviations detected during performance of a procedure by ahealthcare professional. The notification component 702 can further beconfigured to generate and send a notification to a defined entity(e.g., a supervisor) based on the number, frequency and/or compiledseverity levels of the errors/deviations being above a threshold level.Certain procedures can be marked as high, medium, and low risk asseverity classifications, for example. The higher the severity, the moredamage to patient health or the healthcare institution could occur ifthe procedure is not performed correctly. For example, checking apatient in at the front desk might be a low risk procedure andperforming a heart surgery might be a high risk procedure.

The audio feedback component 704 can be configured to provide feedbackdetermined and/or generated by the feedback component 120 to ahealthcare professional performing a procedure in audio form (inaddition to or in the alternative to an AR form). According to theembodiment, the healthcare professional can wear (e.g., included in theuser device 102) or otherwise be within audible range of a speaker viawhich the audio feedback component 704 can send the audio feedback forplayback. In some implementations, the audio feedback component 704 canprovide the healthcare professional with a defined sound to indicatewhen an error is detected (e.g., an alarm sound). Likewise, the audiofeedback component 704 can also provide the healthcare professional withanother device sound if the healthcare professional performs an aspectof a procedure correctly. In other implementations, the audio feedbackcomponent 704 can generate a spoken rendering of the feedback generatedby the feedback component 120 for playback to the healthcareprofessional as speech. For example, if the healthcare professionalperforms step 2 of a procedure incorrectly, the audio feedback component704 can generate and provide the healthcare professional with a verbalinstruction stating that step 2 was performed incorrectly.

The verbal instruction can also explain the aspect of step 2 that wasperformed incorrectly and further provide instruction regarding how tocorrect the error. Thus while performing the procedure, the healthcareprofessional can listen to the feedback and respond accordingly inreal-time.

The live remote assistance component 706 can be configured to provide ahealthcare professional with live remote assistance during a procedurewhen such assistance is needed or requested. For example, in someembodiments, based on detection of certain errors associated with highseverity levels (e.g., based on a predefined severity classificationsystem), errors above a certain amount, frequency, severity, etc., thelive remote assistance component 706 can be configured to automaticallyconnect the healthcare professional to another person at a remotelocation to help guide the healthcare professional. In otherembodiments, the healthcare physician can initiate and request the liveremote assistance functionality on demand. Using the connection, thehealthcare professional and the remote assisting entity can communicatevia a voice and/or voice and video media session. The live remoteassisting entity can further provide the healthcare professional withlive assistance during the procedure. In some implementations, theremote assisting entity can be provided with a live visual feed (e.g.,capture via camera 104 or another camera in the procedure environment)of the procedure and/or the procedural environment to facilitateassisting the healthcare professional. For example, in someimplementations, the user device 102 can include a phone or phoneapplication with voice and/or video capabilities. According to thisexample, the live remote assistance component 706 can automatically acall to the user's phone or phone application that connects thehealthcare professional to the remote assisting entity. In anotherexample implementation, the call or live media session can beestablished between the remote assisting entity and another device inthe procedure environment that can be seen and/or heard by thehealthcare professional.

FIG. 8 illustrates a block diagram of an example user device (e.g., userdevice 102) that facilitates providing auxiliary information regardinghealthcare procedure and system performance in real-time using AR inaccordance with one or more embodiments described herein. Repetitivedescription of like embodiments employed in respective embodiments isomitted for sake of brevity.

User device 102 can include one or more input devices 802, communicationcomponent 810, rendering component 812 and display 106. The user device102 can also include or otherwise be associated with at least one memory818 that stores computer-executable components (e.g., the renderingcomponent 812). The user device 102 can also include or otherwise beassociated with at least one processor 816 that executes thecomputer-executable components stored in the memory 818. The user device800 can further include a device bus 814 that can couple the variouscomponents including, but not limited to, the one or more input devices802, the communication component 810, the rendering component 812, thedisplay 106, the processor 816 and the memory 818.

In one or more embodiments, the user device 102 can capture feedbackregarding performance of a healthcare procedure by a healthcareprofessional and/or regarding a healthcare environment. The user device102 can further provide this feedback to the AR assistance module 110for processing in real-time in association with evaluating performanceof the procedure and/or the environment. For example, the one or moreinput device 802 can include camera 104 which is previously described.The one or more input devices 802 can also include but are not limitedto, an audio capture device 804, one or more motion sensors 806 and/orone or more biometric sensors. The audio capture device 804 can includea microphone or another type of audio capture device that can receiveand record audio during a procedure, such as speech spoken by thehealthcare professional performing the procedure, the patient, and/orone or more other healthcare professionals involved in the procedure. Insome implementations, the audio capture device 804 can further processcaptured audio to convert detected speech to text for providing to theAR assistance module 110. The one or more motion sensors 806 caninclude, for example, an accelerometer and/or a gyroscope that candetect motion of the user device 102 when worn, held or otherwiseoperated by the user. The one or more biometric sensors 808 can includebiometric sensors that detect biometric information for the user duringperformance of a procedure, including but not limited to, heart rate,respiratory rate, stress hormone levels of the user, and the like. Insome implementations, one or more of the motion sensors 806 and/or thebiometric sensors 808 can be external to the user device 102 (e.g., wornby a user, implanted within a user, etc.) and communicatively coupled tothe user device 102.

The communication component 810 can facilitate wired and/or wirelesscommunication between the user device 102 and the server device 108, theone or more external information sources 138 and/or another suitableexternal device (not shown). For example, the communication component310 can receive real-time feedback information from the AR assistancemodule (e.g., via the AR component 128, the notification component 702,the audio feedback component 704, and/or the live remote assistancecomponent 706) regarding performance of a procedure or proceduralenvironment. The communication component 310 can be or include hardware(e.g., a central processing unit (CPU), a transceiver, a decoder),software (e.g., a set of threads, a set of processes, software inexecution) or a combination of hardware and software that facilitatesthe various type of wireless communicating information of informationdescribed herein.

The rendering component 812 can be configured to render the feedbackreceived from the AR assistance module 110 in real-time. For example, inan embodiment in which the auxiliary information includes audio, therendering component 812 can cause the audio to be played at the userdevice 102 via a speaker of the user device (e.g., not shown). Inanother example, in an embodiment in which the feedback includes visualoverlay data provided by the AR component 128 (e.g., text, symbols,icons, images, video, hyperlinks, etc.), the rendering component 812 cangenerate a graphical user interface that can be displayed via thedisplay 106 of the user device 102. The graphical user interface caninclude the overlay data overlaid onto a view of the user's environment.In some implementations, the view includes a real direct view of theenvironment as viewed through a transparent display (e.g., glasses,goggles, contact lenses, etc). In other implementations, the view caninclude live or recorded video/image data of the environment renderedfrom a current perspective of the user relative to the environment. Inanother example implementation, the view can include a 2D or 3D model ofthe environment rendered from the current perspective of the userrelative to the environment.

The manner in which the overlay data is rendered in the graphical userinterface can vary. For example, as exemplified in FIGS. 2-4, theoverlay data can be displayed as text and/or symbols included insemi-transparent pop-up display window or frame. In someimplementations, the overlay data can include visual data thathighlights (e.g., in a distinguishing color), or otherwise drawsattention to a particular object viewed by the user via the display. Forexample, as exemplified in FIG. 2, the overlay data can include colordata that highlights the appearance of the IV fluid bag. In someembodiments, the overlay data can be selected or otherwise interactedwith (e.g., to follow a hyperlink represented in the overlay data, toplay a video represented in the overlay data, and the like).

In one or more embodiments, the rendering component 812 can render theoverlay data such that it is spatially aligned with one or more definedobjects included in the view of the user's environment viewed through oron the display 106 (e.g., a real direct view or an indirect viewconsisting of image/video data or model data). According to theseembodiments, the one or more objects for spatially aligning the overlaydata with can be determined by the AR component 128 and provided to therendering component 812 with the overlay data (e.g., as metadata). Forexample, the AR component 128 can provide the rending component 812 withoverlay data as well as information indicating the overlay data shouldbe aligned with a certain object, such as the “incision site.” Based onthe information identifying the object, in some implementations, usingimage analysis techniques, the rendering component 812 can identify theincision site in the image data of the environment corresponding to acurrent perspective of the user relative to the environment. Forexample, the image data can include a still image of the environment,live video data of the environment, or model data of the environment asviewed from the current perspective of the user to the environment. Therendering component 812 can further determine the relative spatiallocation of the object in the image data to the user, including therelative (3D) location of the object to the user (e.g., based depthinformation associated with the object and a known position/orientationof the user). In other embodiments, the rendering component 812 canreceive information from the procedure characterization component 114identifying the relative spatial location of the object to the user incorresponding image data. The rendering component 812 can further renderthe overlay data on the display 106 based on the relative spatiallocation of the object in the image data and the current perspective ofthe user relative to the object such that it appears to be spatiallyaligned on or near the object.

FIG. 9 illustrates a block diagram of another example user device (e.g.,user device 102) that facilitates providing auxiliary informationregarding healthcare procedure and system performance in real-time usingAR in accordance with one or more alternative embodiments describedherein. In particular, in the embodiment shown, the user device 102includes the AR assistance module 110. The user device 102 also includethe procedure guideline/protocol information 132 in memory 818.According to this embodiment, the user device 102 can perform thevarious processing functions provided by the AR assistance module 110without communicating with the sever device 108. The user device 102 canfurther communicate with one or more external information sources 138 toaccess information as needed. Repetitive description of like embodimentsemployed in respective embodiments is omitted for sake of brevity.

Referring now to FIG. 10, presented is a block diagram of an examplesystem 1000 that facilitates providing auxiliary information regardinghealthcare system performance in real-time using AR in accordance withone or more embodiments described herein. System 1000 can include sameor similar features and functionality as systems 100, 600 and 700.However, system 1000 adds some additional components to the ARassistance module 110 that facilitate additional and/or alternativefunctions of the AR assistance module 110 related to providing a userauxiliary information in real-time regarding the utilization andperformance of physical elements of a healthcare facility that arecurrently being viewed by the user. These additional components includeenvironment recognition component 1002 and an associated environmentcharacterization component 1004, environment assessment component 1006and associated equipment/supplies assessment component 1008, selectioncomponent 1016, and virtual reality (VR) component 1018. Theseadditional components also include some new sub-components of thefeedback component 120, including descriptive feedback component 1010,utilization feedback component 1012, and performance feedback component1014. Repetitive description of same or similar functionalities of likecomponents is omitted for sake of brevity.

The AR assistance module 110 can be configured to perform various ARtype services associated with facilitating healthcare professionalsperforming their jobs. As described with reference to systems 100, 600and 700, in various embodiments, the AR assistance module 110 can beconfigured to assist healthcare professionals when performing a definedhealthcare procedure by providing the healthcare professional withauxiliary information, determined in real-time, that is relevant to acurrent aspect of the procedure being performed.

In various additional embodiments, as exemplified in system 1000 (andadditional systems 1300 and 1600 described supra), the AR assistancemodule 110 can be configured to assist healthcare administrators (orother type of healthcare organization employee) in understanding andassessing various aspects of a state of performance of a healthcareorganization by providing the healthcare administrators with auxiliaryinformation associated with a current physical area of a healthcarefacility of the organization the healthcare professional is viewing orotherwise examining For example, the healthcare organization can includebut is not limited to: a hospital network, a hospital, a nursing home,an assisted living facility, a physical therapy facility, a pharmacy, aclinic, a medical imaging facility, a laboratory, and the like. Thedifferent areas of the healthcare facility will vary based on the typeof healthcare facility. For example, with respect to a hospital, thedifferent areas may be associated with different sectors or departmentsof the hospital that provide different medical services (e.g.,emergency, surgery, pediatrics, internal diseases, gynecology, nursing,laboratory, imaging, etc.), and different rooms/areas in the respectivesectors or departments (e.g., patient waiting room, patient examinationroom, clerical room, supplies room, etc.).

In one or more embodiments, the environment recognition component 1002can be configured to determine an area or environment of a healthcarefacility that user is currently viewing or would like to view and theenvironment characterization component 1004 can be configured tocharacterize various aspects of the environment or area. In this regard,the environment characterization component 1004 can be configured toidentify physical objects (e.g., equipment, supplies, furniture, people,etc.) included in the area. The environment characterization component1004 can further be configured to determine the relative 2D and/or 3Dspatial locations of the objects to one another and a user viewing thearea based on the current perspective of the user.

In some embodiments, the environment recognition component 1002 candetermine the area/environment based on received image data (e.g., videoand/or still images) captured of the area/environment. For example, inone implementation, a user can wear or hold the user device 102 as theuser moves throughout a healthcare facility and capture image data(e.g., video and/or still images) of the healthcare facility via thecamera 104 from the perspective or the user. The user device 102 canfurther provided the image data to the AR assistance module 110 forprocessing thereof in real-time. According to this implementation, usingone or more image analysis techniques, the environment characterizationcomponent 1004 can identify one or more objects and features of acurrent area or environment being viewed by the user. For example, theenvironment characterization component 1004 can identify objects (e.g.,equipment, supplies, furniture, people) included in the image data usingpattern recognition techniques or other image analysis techniques. Theenvironment characterization component 1004 can further be configured todetermine the relative 2D and/or 3D spatial locations of the objects toone another and a user viewing the area based on the current perspectiveof the user. For example, in some implementations, the image data can beassociated with depth or distance information for respective objects(i.e., groups of identified pixels forming the respective objects)included therein. This depth information be can be captured via thecamera that captures the image data and/or determined using stereoscopyanalysis of partially overlapping images. The environmentcharacterization component 1004 can further determine the spatiallocations of the objects based on the depth or distance information. Insome implementations, the environment recognition component 1002 canfurther access indexed information (e.g., stored in memory 130, one ormore external information sources 138, or at another device) thatassociates the one or more objects and/or patterns in the image data(e.g., regarding relative positions of the objects and the types ofobjects) with defined physical locations or positions (e.g., camerapositions/perspectives) in the healthcare facility. The environmentrecognition component 1002 can then determine the location and/orcurrent position/perspective of the area of the environment being viewedbased on comparison of the detected objects and/or patterns in thedetected objects with the indexed information.

In another implementation, the environment recognition component 1002can receive image data (e.g., video and/or still images) of a user asthe user moves about a healthcare facility. For example, the image datacan be captured via cameras dispersed at various locations throughoutthe healthcare facility and provided to the AR assistance module 110 inreal-time for processing. According to this implementation, using one ormore image analysis techniques, the environment characterizationcomponent 1004 can identify the user and track the user as the usermoves throughout the environment. Using one or more image based analysistechniques, the environment characterization component 1004 can furtheridentify objects included in a current area or environment being viewedby the user as the user moves about the healthcare facility. Forexample, the environment characterization component 1004 can identifyobjects (e.g., equipment, supplies, furniture, people), and relativelocations of the objects in the image data that appear within adetermined field of view of the user. In some implementations, theenvironment recognition component 1002 can further access indexedinformation (e.g., stored in memory 130, one or more externalinformation sources 138, or at another device) that associates the oneor more objects and/or patterns in the image data with defined physicallocations or positions (e.g., camera positions/perspectives) in thehealthcare facility. The environment recognition component 1002 can thendetermine the location and/or current position/perspective of the areaof the environment being viewed based on comparison of the detectedobjects and/or patterns in the detected objects with the indexedinformation.

Various additional techniques can be used to determine a currentposition of a user relative to various areas and associated objects in ahealthcare facility as the user moves about the healthcare facility. Forexample, in embodiments in which the user wears or holds the user device102 as the user moves about the facility, the user device 102 can beconfigured to determine the current location of the user using variouslocation based detection methods (e.g., GPS, triangulation, beacons,detection of RFID tags associated with defined location informationand/or perspectives of the facility). In another example, the contextcomponent 118 can determine the location of the user by evaluatingmovement of the user based on image data captured of the user and/orfrom the user's perspective (e.g., via camera 104 in implementations inwhich the user wears or holds the user device). Similarly, in someimplementations, the user device can include one or more motion sensorsconfigured to generate motion data as the user moves about thehealthcare facility and/or looks in different directions/orientationswith the user device (e.g., when the user device 102 includes an eyeweardevice). The context component 118 and/or the user device can furtherevaluate the motion data in real-time to determine the current locationand/or perspective of the user based on the motion data (e.g., includingmovement and orientation information).

In some implementations, based on determination of a current locationand/or perspective of a user relative to an area/environment of ahealthcare facility (e.g., determined using GPS data, triangulation,RFID tags, motion information, etc.), the environment characterizationcomponent 1004 can determine objects that are associated with thecurrent location and perspective of the user using indexed informationthat associates known objects with the current location/perspective.Still in other embodiments, the user device 102 can be configured toidentify objects that are in line of site and/or a current field of viewof the user based on image data captured by the camera 104 and a knowposition/orientation of the user. The user device 102 can furtherprovide the AR assistance module 110 with object information identifyingthe objects and the relative positions of the objects to the user basedon the user's current position/orientation.

In another implementation, the user can be located at a differentlocation relative to the area or environment of the healthcare facilitythat the user desires to view. For example, using a device (e.g., userdevice 102) that is located a remote location, the user can access andview image data (e.g., video and/or still image data) of a selected areaof the healthcare facility. For example, the user can provide inputindicating an area of the healthcare facility the user would like toview (e.g., such as input requesting a view of operating room B). Basedon received user input indicating a desired area and/or view of thehealthcare facility, the user device 102 can be configured to render(e.g., via rendering component 812) image data of the desired area. Theimage data can include live video data being captured of the selectedarea, still image data captured of the selected area, or model data(e.g., 2D/3D model data) including a regenerated representations ormodels of the selected area. With this implementation, the particulararea and/or view of the environment will be known to the environmentrecognition component 1002 based on the input selection made by theuser. In some one or more embodiments, the environment characterizationcomponent 1004 can use image analysis to identify objects and relativelocations of the objects in the image data that is rendered to the user.For example, in implementations in which live video data of the selectedlocation is rendered to the user, the environment characterizationcomponent 1004 can analyze the image data to identify objects andrelative locations of the objects in the image data. In otherembodiments, the environment characterization component 1004 can accessindexed information (e.g., in memory 130, at one or more externalinformation sources 138, or at another device), that associates knownobjects with respective areas in the healthcare facility. Based on theselected area, the environment characterization component 1004 canemploy the indexed information to identify objects that are included inthe selected area. In some implementations, the environmentcharacterization component 1004 can used this indexed information toassist with identifying objects included in the rendered image datausing one or more image based analysis techniques.

The environment assessment component 1006 can be configured to assess acurrent area of a healthcare facility that is viewed (or has beenselected for view) by a user based on the information determined,generated and/or received by the environment recognition component 1002identifying the current area and/or information determined, generated,and/or received by the environment characterization component 1004identifying objects and relative locations of the objects included inthe user's current view. The environment assessment component 1006 canbe configured to determine a variety of information associated with thearea and the one or more objects included in the view of the area.

In implementations in which the one or more objects include equipment orsupplies, the environment assessment component 1006 can employ theequipment/supplies assessment component 1008 to determine various typesof equipment/supplies information associated with the equipment orsupplies. For example, the equipment can include medical equipment(e.g., imaging machines, anesthesia machines, etc.) and non-medicalequipment (e.g., computers used for clerical purposes, maintenanceequipment, etc.). The supplies can include any type of medical supply,for example tools or instruments used for performing a procedure (e.g.,surgical supplies), laboratory supplies, pharmaceuticals, and the like.In some implementations, the supplies can also included non-medicalsupplies (e.g., office supplies). The equipment/supplies information caninclude but is not limited to: descriptive information that describesspecifications of the equipment or supplies, utilization informationregarding utilization of the equipment or supplies, and performanceinformation regarding clinical and/or financial performance of thehealthcare facility associated with the equipment or supplies. Forexample, descriptive information associated with a particular piece ofmedical equipment or supply can include the type of information thatwould be likely be associated with the equipment or supply on a salesspecification, such as but not limited to: a name or title for theequipment or supply, a description of the intended use, a manufacturermake and model, a dimension or sizing information, cost information, andthe like. Utilization information regarding utilization of a particularpiece of medical equipment or supply can include information regardingusage of the medical equipment or supply by the current healthcareorganization. For example, such usage information can describe how thehealthcare facility uses the medical supply or equipment, informationregarding degree of past usage, information regarding frequency of pastusage, information regarding expected usage, and the like. In anotherexample, the usage information can relate to current usage state orstatus of the medical supply or equipment (e.g., used or unused). In yetanother example, the usage information can include information thateffects usage of a medical supply or equipment, such as informationregarding maintenance type issues associated with a piece of medicalequipment or supply.

Performance information regarding clinical performance of the healthcarefacility associated with the equipment or supplies can include forexample information regarding patient success outcome information andquality of care associated with usage of a piece of medical equipment ormedical supply. In some embodiments, this type of performanceinformation can be compared to usage of alternative supplies, such asrelative success rates and relative quality of care. Clinicalperformance information associated with medical supplies can alsoinclude regarding usage and availability of supplies that affect theclinical performance of the healthcare organization. For example, suchinformation can indicate whether there is a surplus or deficiency inamount of the supply available to meet patient needs. In anotherexample, such information can indicate whether a supply or tool was usedproperly, disposed of properly, cleaned properly, stored properly andthe like. Likewise, performance information regarding financialperformance of the healthcare facility associated with equipment orsupplies can include financial gain and loss information attributed tousage of a piece of medical equipment or medical supply.

In various embodiments, the descriptive, utilization and/or performanceinformation that is associated with the equipment and suppliesidentified by the environment characterization component 1004 can bepredetermined and included in one or more databases accessible to the ARassistance module 110 (e.g., in memory 130, at one or more externalinformation sources 138, or at another device). For example, suchinformation can have been previously generated in various inventorylogs, device logs, purchase reports, maintenance reports, andperformance evaluation reports/logs, billing reports, and the like, andassociated with the respective supplies and/or equipment. Usually, suchdevice logs and error logs are reviewed by only folks with deeptechnical know-how. With the system and methods contemplated herein,system 100 can find relevant information and make it easy to view andunderstand for non-technical folks such as clinicians and hospitaladministrators.

According to these embodiments, the equipment/supplies assessmentcomponent 1008 can scan various databases and sources to find andretrieve such information associated with a supply or medical equipmentthat is included in a user's current view. In other implementations, theequipment/supplies assessment component 1008 can run real-time reportson the data included in the various databases and sources to generatethe utilization and performance information in real-time. For example,with respect to financial performance information associated withequipment and supplies included in an area of a healthcare facilityviewed by a user, the equipment/supplies assessment component 1008 canaccess purchase information, usage information, maintenance information,billing information, etc. associated with usage and performance of therespective supplies and equipment and determine monetary valuesindicating costs associated with the respective supplies and equipment.For example, the cost information can indicate costs to the healthcareorganization attributed to purchasing, using and maintaining therespective supplies and equipment and/or ROI associated with thesupplies and/or equipment. The equipment/supplies assessment component1008 can also access information regarding expected and/or budgetedcosts for the equipment and supplies and determine information regardinga degree to which the actual costs associated with certain supplies andequipment are over budget or under budget.

The feedback component 120 can be configured to collate the informationretrieved, generated or determined by the environment assessmentcomponent 1006 for potential provision as auxiliary information to theuser viewing the assessed area as feedback in real-time. For example,the descriptive feedback component 1010 can collect and collatedescriptive information generated by the equipment/supplies assessmentcomponent 1008 that describes specifications of the equipment orsupplies. The utilization feedback component 1012 can collect andcollate utilization information generated by the equipment/suppliesassessment component 1008 regarding utilization of the equipment orsupplies, and the performance feedback component 1014 can collect andcollate performance information generated by the equipment/suppliesassessment component 1008 regarding clinical and/or financialperformance of the healthcare facility associated with the equipment orsupplies. Such collection and collation of the descriptive, utilization,and performance information can include cleaning and indexing theinformation based on the respective objects (e.g., equipment or supply)each piece of information is associated with. In various implements, thefeedback component 120 can associate metadata with each piece ofinformation identifying the particular object that it is associatedwith.

Given the amount of potential feedback information that can be gatheredby the feedback component 120 regarding an area of a healthcare facilitythat is being viewed by a user, in many scenarios it may not be feasibleor useful to provide the user with all the feedback information. Forexample, with respect to supplies and hardware identified, a user mayonly be interested in receiving financial performance informationassociated with a single piece of equipment in the room. The in variousimplementations, the amount and type of feedback information that isprovided to a particular user can be filtered and selected by theselection component 1016 based on a current context of the user. Inparticular, the selection component 1016 can select a subset of thefeedback information (e.g., generated/collated by the feedback component120) for provision to a user viewing an area of a healthcare facilitybased on a current context of the user. In some implementations, thecurrent context of the user can be determined by the context component118 (e.g., based on information included in memory or one or moreexternal information sources 138, based on received informationregarding movement of the user, and the like). In this regard, thecurrent context can include but is not limited to: a role of the user atthe healthcare facility (e.g., job title and description), authorizationof the user to receive various types of information (e.g., the CEO ismore privy to certain financial information relative to entry levelsecretary), or a purpose of the user for viewing a particular area(e.g., to gain financial information, to evaluate sanitation andmaintenance, to perform maintenance duties, etc.). For example, theauxiliary information that is provided to a maintenance worker canreflect maintenance issues associated with equipment in an observed areawhile the auxiliary information that is provided to a finance executivecan concern cost and ROI performance data associated with equipmentand/or medical services supported by the area.

In some implementations, the context can also include a degree ofmobility of the user. According to this implementation, the amount ofinformation that should be displayed for each object a user views shouldreflect the amount of time the user will be viewing the object and theability of the user to read or comprehend the amount of informationprovided. For example, if the user is walking through a hospital, theuser can pass by many different objects quickly and thus not have enoughtime to read detailed information about each object. However, if a useris standing still and looking at an area of the healthcare facility, theuser can be provided with more information about the respective objectsin area.

Similar to the feedback information provided to user in association withperformance of a procedure, in some implementations the AR component 128can provide the user with feedback information regarding the state ofperformance of the healthcare organization using an AR experience.According to these implementations, the AR component 128 can generateoverlay data that comprises or represents the subset of the informationselected by the selection component 1016 for providing to the user. Forexample, using an AR device (e.g., user device 102), a user canphysically visit an area of a healthcare facility and receive overlaydata concerning the performance of the area, including equipment andsupplies associated with the area, people/employees associated with thearea, and the like. The overlay data can be rendered to the user via thedisplay of the AR device as over a current view of the area viewedthrough or on the display. In an additional implementation, the user canremotely view the area by viewing video and/or image data captured ofthe area. According to this implementation, the overlay data can beoverlaid onto the video and/or image data of the area that is displayedon the user's remote device. Repetitive description of the manner inwhich the AR component 128 generates and provides overlay data forpresenting to a user is omitted for sake of brevity.

The ability to be physically situated near real world healthcare objectsand have performance and financial data be overlaid on a physical viewof the real world healthcare objects is valuable to clinicians andhospital administrators. They can have a more real and tangibleunderstanding of how their healthcare institution is working, includingsuccesses and areas for improvement.

In another implementation, the user can view the area using a virtualreality (VR) system/device. According to this implementation the ARassistance module 110 can include VR component 1018 to facilitategenerating overlay data for rendering to a user in a VR experience. Forexample, using a VR device (e.g., user device 102), a user can bepresented with a 3D space model/visualization of the area that appearsto the user as if the user is actually standing in the 3D space model.The auxiliary data regarding the area can be projected onto or otherwiseintegrated within the 3D space model view of the area. The mechanism viawhich the overlay data is included in a VR visualization of an area of ahealthcare facility can be similar to that used for integrating overlaydata on a real view of an area. However in some implementations, theoverlay data provided in a VR experience can be 3D object and interactedwith by the user as if it were a object the use can hold, touch, move,etc. In various embodiments, regardless as to whether the user isviewing the auxiliary data using an AR device or a VR device, theauxiliary data can be spatially aligned with respective objects, people,equipment/supplies, etc., that is associated with. For example,auxiliary data concerning how a supply cabinet in a room is understocked with certain supplies can be overlaid on a direct view of thesupply cabinet (e.g., the supply cabinet viewed through a transparentdisplay, such as AR glasses or goggles), or an indirect view of thesupply cabinet (e.g., video/image data or 3D model data of the supplycabinet presented on a display).

In one or more additional embodiments, system 1000 can facilitateproviding a user with AR or VR feedback information regarding equipmentand supplies associated with a location in a healthcare environmentbased on explicit input received from the user. For example, in variousimplementations, the selection component 1016 provides for automaticallyselecting the appropriate AR or VR feedback to provide to a user basedon the user's context (e.g., the user's role or job title, the user'spurpose for visiting the location, the user's mobility state, etc.). Inother embodiments, the user can provide the AR assistance module withdirect input indicating, requesting or selecting a particular type offeedback to be provided to the user via AR or VR. For example, the usercan provide input via the user device 102 using a suitable inputmechanism (e.g., a keyboard, a keypad, a touch-screen, gesture basedinput, voice recognition based input, etc.), identifying an object andrequesting information about that object. In another example, the usercan pose a question asking for information about a location or room in ahealthcare facility such as requesting information about where certainsupplies are located in the room. In another example, the user canprovide input asking for a certain type of information about a locationin the healthcare facility. According to this example, the user may sayfor instance, “show me supplies that are low in stock,” “show me theequipment having maintenance issues,” “show me the part of this imagingdevice that is not working properly,” etc. This input can be received bythe environment assessment component 1006 and/or the feedback components120. Based on reception of the user input and other input such as theworker's current location, the environment assessment component 1006 candetermine the answer to the user's question/request using the mechanismsdescribed herein. The feedback component can further generate feedbackinformation that answers the user's question or responds to the user'srequest and the AR component 128 can further provide the user with thefeedback using AR or VR.

For example, a new maintenance worker wearing an AR device (e.g., userdevice 102) including a voice recognition input mechanism may walk intoa room and provide verbal input asking where the non-latex gloves arelocated. The equipment assessment component 1006 can then determinewhere the non-latex gloves are located in the room, and the AR component128 can provide the user with feedback in the form of an overlay thatnotes the particular location in the room including the non-latexgloves. For example, if the non-latex gloves are located on the leftside second shelf of the supply cabinet, the overlay data can highlight,mark, or otherwise draw attention to the left side second shelf of thesupply cabinet. In some implementations, the feedback component 120 canalso provide the user with audio feedback instructing the worker wherethe non-latex gloves are located.

FIGS. 11, 12A and 12B presents example AR visualizations 1100 and 1200respectively including auxiliary information regarding usage and/orperformance of a healthcare system equipment in accordance with one ormore embodiments described herein. One or more aspects of the example ARvisualizations 1100 and 1200 demonstrate the features andfunctionalities of system 1000 (and additional systems describedherein). Visualizations 1100 and 1200 respectively depict an area of ahealthcare facility that is being viewed by a user. In the embodimentsshown, the user 1102 is wearing an AR device 1104 and physicallystanding in the healthcare facility with a direct view of the area ofthe healthcare facility viewed through transparent display of the ARdevice 1104. However in other implementations, the user 1102 can beprovided at a remote location and view image/video data of the areaand/or model data of the area on a remote device. In variousembodiments, the AR device 1104 can be or include user device 102.Further, the AR device 1104 can include or be communicatively coupled tothe AR assistance module 110 to facilitate providing the user withauxiliary information regarding usage and/or performance of a healthcaresystem equipment in association with viewing the equipment.

With reference to FIG. 11, in visualization 1100, the user 1102 isviewing an imaging room of a healthcare facility that includes animaging device 1106 and an associated computing system 1108.Visualization 1100 further includes overlay data associated with theimaging device 1106, the computing system 1108, and the imaging room ingeneral, regarding utilization and/or performance of the imaging roomand associated equipment. For example, the overlay data includes adisplay window 1110 associated with the imaging device 1106. The displaywindow 1110 includes text information identifying the imaging device asa Super CT (computed tomography) machine and indicating that it has an87% uptime. The overlay data also includes display window 1112 with textindicating that the power distribution unit (PDU) has had two failuresthis year. The overlay data further includes display window 1114associated with the imaging room in general with text indicating thathigh radiation has been detected. This can be detected by radiationsensors in the room or in the CT machine itself.

With reference to FIG. 12A, in visualization 1200, the user 1102 is alsoviewing an imaging room of a healthcare facility that includes animaging device 1202. Visualization 1200 further includes overlay dataassociated with the imaging device 1202 that includes a display window1204 with text indicating the imaging device has a maintenance issuethat involves a broken tube. The text also provides additionalinformation regarding a scheduled maintenance appointment for theimaging device 1202. The display window 1204 also includes a symbol andimage associated with the text. The overlay data further includes agraphical depiction 1206 of the particular part of the imaging devicethat is broken.

With reference to FIGS. 10, 11, and 12A in accordance with theembodiment shown, the overlay data (e.g., display windows 1110, 1112,1114, 1204 and graphical depiction 1206) was generated by the ARcomponent 128 based on feedback, selected from selection component 1016regarding equipment description, utilization and/or performanceinformation that was relevant to the user 1102. In one or moreembodiments, the feedback was determined and/or generated by thefeedback component 120 (e.g., via descriptive feedback component 1010,utilization feedback component 1012, and/or performance feedbackcomponent 1014), in accordance with the techniques described withreference to system 1000. The overlay data was further rendered on thedisplay of the AR device 1104 over a direct view of the imaging roomequipment viewed by the user 1102 through a transparent display of theAR device 1104.

It should be appreciated that the appearance and location of the overlaydata (e.g., display windows 1110, 1112 and 1114) in visualizations 1100and 1200 is merely exemplary and intended to convey the concept of whatis actually viewed by the user through the AR device 1104. However, theappearance and location of the overlay data in visualizations 1100 and1200 is not technically accurate, as the actual location of the overlaydata would be on the glass/display of the AR device 1104. For example,FIG. 12B depicts another perspective of visualization 1200 as viewedthrough the lenses of the AR device 1104.

FIG. 13 illustrates a block diagram of another example system 1300 thatfacilitates providing auxiliary information regarding healthcare systemperformance in real-time using AR in accordance with one or moreembodiments described herein. System 1300 includes same or similarfeatures and functionalities as system 1000 with the addition ofemployee assessment component 1302 to the environment assessmentcomponent 1006. Repetitive description of same or similarfunctionalities of like components described in prior embodiments isomitted for sake of brevity.

In addition to assessing equipment and supplies included in an area of ahealthcare facility that is viewed by a user, the environment assessmentcomponent 1006 can include employee assessment component 1302 to assessemployees of the healthcare organization that are included in the area.For example, in implementations in which a user is viewing a room at ahealthcare facility (e.g., an operating room), the employee assessmentcomponent 1302 can determine information regarding the respectiveemployees in the room. In another example, in implementations in which auser is walking through various areas in a healthcare facility, theemployee assessment component 1302 can determine information regardingrespective employees the user passes by or encounters. In anotherexample, the environment characterization component 1004 can identify anemployee that the user is looking at (e.g., within a direct line of siteof the user), and determine information regarding the specific employeethe user is looking at.

The employee assessment component 1302 can determine a variety ofinformation about employees identified by the environment recognitioncomponent 1002. For example, the employee information can include but isnot limited to, descriptive information that describes various factsabout the employee, utilization information regarding how the employeeis used by the healthcare facility, and performance informationregarding the employee's evaluating the employee's performance at thehealthcare organization. For example, the descriptive information couldinclude information can include information that identifies the employee(i.e., the employee's name), identifies the job title of the employee,identifies the qualifications of the employee, identifies the durationof time the employee has worked for the healthcare organization,identifies the employee's salary, and the like. In another example, thedescriptive information can include more personal information regardingthe employee's preferences, family, hobbies, and the like. The usageinformation about an employee could include for example, informationregarding the employee's duties at the healthcare organization,information regarding the employee's schedule, information regardingfrequency and amount of such duties performed (e.g., number of X typeprocedures performed per week), and the like. Performance informationassociated with an employee can include information regarding how wellthe employee is performing his or her duties at the healthcareorganization and as well as financial and/or clinical performance of thehealthcare organization attributed to the employee. For example,performance information associated with a physician can includeinformation regarding patient outcomes and satisfaction, efficiency withrespect to performing his or her duties, ROI attributed to thephysician's performance at the healthcare organization, and the like.

In various embodiments, the descriptive, utilization and/or performanceinformation that is associated with respective employees identified bythe environment characterization component 1004 can be predetermined andincluded in one or more databases accessible to the AR assistance module110 (e.g., in memory 130, at one or more external information sources138, or at another device). According to these embodiments, theequipment/supplies assessment component 1008 can scan various databasesand sources to find and retrieve such information associated with asupply or medical equipment that is included in a user's current view.For example, the one or more external information sources 138 caninclude employee information identifying all healthcare professionalemployees of a healthcare organization that employs the AR assistancemodule 110 to facilitate monitoring and improving the performance of thehealthcare organization. For example, the employee information canidentify the respective healthcare professional employees, their roles(e.g., job title), the patients they work with (if having a clinicianrole), the procedures they are authorized to perform, the proceduresthey are expected to perform, and the like. The employee information canalso include any possible performance information that can be monitoredand recorded for the employee over time, such as hours worked,procedures performed, outcomes of the procedures, evaluations of theprocedures, satisfaction of patients, reviews of the employee,information regarding how the employee performs certain tasks relativeto other employees or another benchmark, and the like. The employeeinformation can also include financial information regarding salary ofthe respective employees, whether the employee has entered overtimehours, return on investment (ROI) generated from the respectiveemployees, and the like. The employee information can also includepersonal information associated with the respective employees, such asinformation regarding an employee's family and friends, preferences,birthday, hobbies, awards, vacations, and the like. The employeeinformation can be updated on a regular basis (e.g., hourly, daily,weekly, etc.) such that the information source provides currentinformation regarding various aspects of an employee's work life andpersonal life. In other implementations, the employee assessmentcomponent 1302 can run real-time reports on the data included in thevarious databases and sources to generate the utilization andperformance information in real-time.

The employee assessment component 1302 can further be configured todetermine information about other people identified in an area of ahealthcare facility that are not employees. For example, the employeeassessment component 1302 can determine information associated withpatients, and visitors of the healthcare facility. Such information canalso be found in one or more external information sources 138 (e.g., inpatient information and records).

The feedback component 120 can be configured to collate the informationretrieved, generated or determined by the employee assessment component1302 for potential provision as auxiliary information to the userviewing the assessed area as feedback in real-time. For example, thedescriptive feedback component 1010 can collect and collate descriptiveinformation generated by the employee assessment component 1302 thatdescribes the employees. The utilization feedback component 1012 cancollect and collate utilization information generated by the employeeassessment component 1302 regarding how the healthcare organization usesthe respective employees, and the performance feedback component 1014can collect and collate performance information generated by theemployee assessment component 1302 regarding clinical and/or financialperformance of the healthcare facility associated with employees. Suchcollection and collation of the descriptive, utilization, andperformance information can include cleaning and indexing theinformation based on the respective employee each piece of informationis associated with. In various implements, the feedback component 120can associate metadata with each piece of information identifying theparticular employee that it is associated with.

Given the amount of potential feedback information that can be gatheredby the feedback component 120 regarding employees viewed by user, it maynot be feasible or useful to provide the user with all the feedbackinformation for each employee. For example, in some scenarios, the usermay only be interested in information about a particular employeeincluded in an area contacting several employees. In another example,the user may only be interested in receiving performance informationassociated with an employee as opposed to general descriptiveinformation. The in various implementations, the amount and type offeedback information that is provided to a particular user regardingemployees can be filtered and selected by the selection component 1016based on a current context of the user. In particular, the selectioncomponent 1016 can select a subset of the feedback information (e.g.,generated/collated by the feedback component 120) for provision to auser viewing an area of a healthcare facility based on a current contextof the user.

In some implementations, the current context of the user can bedetermined by the context component 118 (e.g., based on informationincluded in memory or one or more external information sources 138,based on received information regarding movement of the user, and thelike). In this regard, the current context can include but is notlimited to: a role of the user at the healthcare facility (e.g., jobtitle and description), authorization of the user to receive varioustypes of information (e.g., the CEO is more privy to certain financialinformation relative to entry level secretary), or a purpose of the userfor viewing a particular area (e.g., to gain financial information, toevaluate performance associated with a specific task, etc.). Forexample, in an implementation in which a user is investigatingperformance of employees associated with billing issues, the selectioncomponent 1016 can be configured to select information to render aboutemployees that is related to billing, such as information regarding howan employee performs at entering billing codes, number of insuranceclaims filed, number of claims rejected, and the like. The context ofthe user can also include information regarding an employee the user iscurrently looking at. According to this implementation, the selectioncomponent 1016 can be configured to select information regarding onlythe particular employee the user is looking at. The context of the usercan also relate to the relationship between the user and the one or moreemployees included in an area viewed by the user. For example, theinformation provided to a user that is a supervisor of an employee wouldlikely be different than the information provided to a user that is asubordinate of the employee. The context of a user can also include adegree of mobility of the user. For example, if the user is merelypassing by an employee that the user is not well acquainted with, theuser can be provided with information reminding the user of theemployee's name and prompting the user to congratulate the employee onhis or her promotion. Alternatively, if the user is meeting with anemployee to discuss improving the employee's performance, the user canbe provided with auxiliary information that highlights some importantpast performance statistics.

Similar to the feedback information provided to user in association withequipment and supplies, in some implementations the AR component 128 canprovide the user with employee feedback information using an ARexperience. According to these implementations, the AR component 128 cangenerate overlay data that comprises or represents the subset of theemployee feedback information selected by the selection component 1016for providing to the user. In another implementation, the VR component1018 to generate overlay data for rendering to a user in a VRexperience. With respect to rendering the overlay data in either AR orVR, in one or more embodiments, the user device (e.g., via renderingcomponent 812), can be configured to associated the overlay data atpositions that coincide with the respective employees that it isassociated with.

In large healthcare institutions with hundreds or thousands of employeesand even more patients, the ability to provide team members andleadership with quick understanding and information about theircolleagues has great value to growing the team and overall employeemanagement. Relationships can be enhanced through such technology aswell as improved employee coaching.

FIG. 14 presents an example AR visualization 1400 including auxiliaryinformation regarding employees of a healthcare organization inaccordance with one or more embodiments described herein. One or moreaspects of the example AR visualization 1400 demonstrate the featuresand functionalities of system 1300 (and additional systems describedherein) with respect to employee assessment component 1302.Visualization 1400 depicts an area of a healthcare facility that isbeing viewed by a user 1102. The area includes four people, twophysicians Anne and Bob, and two administrative personnel. In theembodiments shown, the user 1102 is wearing an AR device 1104 andphysically standing in the healthcare facility with a direct view of thearea of the healthcare facility viewed through transparent display ofthe AR device 1104. However in other implementations, the user 1102 canbe provided at a remote location and view image/video data of the areaand/or model data of the area on a remote device. In variousembodiments, the AR device 1104 can be or include user device 102.Further, the AR device 1104 can include or be communicatively coupled tothe AR assistance module 110 to facilitate providing the user withauxiliary information regarding usage and/or performance of a healthcaresystem equipment in association with viewing the equipment.

Visualization 1400 further includes overlay data associated withphysicians Anne and Bob respectively presented in display windows 1402and 1404. For example, display window 1402 associated with Anne includesa few bullet points with text describing some facts about Anne,including her qualifications, expertise, preferences and performancewith respect to entering payer cods correctly. Likewise, display window1404 associated with bob includes a few more bullet points with textdescribing some facts about Bob, including his qualifications, check intime, salary, experience, HR profile data, usage and daily activitydata, and statistics regarding surgery room set up performance.

With reference to FIGS. 13 and 14 in accordance with the embodimentshown, the overlay data (e.g., display windows 1402 and 1404) wasgenerated by the AR component 128 based on feedback, selected fromselection component 1016 regarding employee description, performance andutilization information that was relevant to the user 1102. In one ormore embodiments, the feedback was determined and/or generated by thefeedback component 120 (e.g., via descriptive feedback component 1010,utilization feedback component 1012, and/or performance feedbackcomponent 1014), in accordance with the techniques described withreference to systems 1000 and 1300. The overlay data was furtherrendered on the display of the AR device 1102 over a direct view of theimaging room equipment viewed by the user 1102 through a transparentdisplay of the AR device 1104. It should be appreciated that theappearance and location of the overlay data (e.g., display windows 1402and 1404) in visualization 1400 is merely exemplary and intended toconvey the concept of what is actually viewed by the user through the ARdevice 1104. However, the appearance and location of the overlay data invisualization 1400 is not technically accurate, as the actual locationof the overlay data would be on the glass/display of the AR device 1104.

FIG. 15 presents an example AR visualization 1500 including auxiliaryinformation regarding various aspects of an operating room environmentin accordance with one or more embodiments described herein. One or moreaspects of the example AR visualization 1500 demonstrate the featuresand functionalities of systems 1000 and 1300 (and additional systemsdescribed herein) with respect to equipment/supplies assessmentcomponent 1008 and employee assessment component 1302.

Visualization 1500 depicts an operating room environment of a healthcarefacility that is being viewed by a user 1102. The environment includesthree physicians operating on a patient. In the embodiments shown, theuser 1102 is wearing an AR device 1104 and physically standing in thehealthcare facility with a direct view of the area of the operating roomenvironment viewed through transparent display of the AR device 1104.However in other implementations, the user 1102 can be provided at aremote location and view image/video data of the area and/or model dataof the area on a remote device. In various embodiments, the AR device1104 can be or include user device 102. Further, the AR device 1104 caninclude or be communicatively coupled to the AR assistance module 110 tofacilitate providing the user with auxiliary information regarding usageand/or performance of a healthcare system equipment in association withviewing the equipment.

Visualization 1500 further includes overlay data comprising informationassociated with various supplies, equipment and people (e.g., thephysicians and the patient) included in the operating room respectivelydetermined by the equipment/supplies assessment component 1008 and theemployee assessment component 1302. The specific information representedin the overlay data includes utilization and performance informationassociated with the various supplies, equipment and people, that theselection component 1016 determined was relevant to the context of theuser 1102. For example, display window 1502 includes supply utilizationinformation regarding gloves and needles in the supply cabinet. Displaywindow 1502 also includes financial performance information regardingcosts attributed to the gloves and needles. Display window 1504 includesinformation regarding costs associated with the trash, includinginformation regarding supply utilization and costs associated therewith.Display window 1506 includes information regarding the surgery beingperformed on the patient, including descriptive information about thesurgery, and financial performance information. Further, the overlaydata includes display windows 1508, 1510, and 1512 respectivelyproviding cost information regarding cost attributed to the utilizationof the respective physicians for the current surgery. As with the othervisualizations described herein, it should be appreciated that theappearance and location of the overlay data (e.g., display windows1502-1512) in visualization 1500 is merely exemplary and intended toconvey the concept of what is actually viewed by the user through the ARdevice 1104. However, the appearance and location of the overlay data invisualization 1500 is not technically accurate, as the actual locationof the overlay data would be on the glass/display of the AR device 1104.

A user, through a user interface should as in-air motion signals, cancontrol the AR to show, edit, and change its display based on theirintentions. A camera can detect such signals and the system 100 canadjust the overlay data accordingly.

FIG. 16 illustrates a block diagram of another example system 1600 thatfacilitates providing auxiliary information regarding healthcare systemperformance in real-time using AR in accordance with one or moreembodiments described herein. System 1600 includes same or similarfeature and functionalities as system 1300 with the addition ofrecommendation component 1602. Repetitive description of same or similarfunctionalities of like components described in prior embodiments isomitted for sake of brevity.

In one or more embodiments, the recommendation component 1602 can beconfigured to analyze the various information/data included in the oneor more external information sources 138 (and/or in memory 130), as wellas the information determined by the environment assessment component1006 over time to determine recommendation information regarding how toimprove an aspect of the state of performance of the healthcareorganization. For example, the recommendation component 1602 canevaluate past performance information regarding the clinical and/orfinancial performance of the healthcare organization associated withutilization of various equipment, supplies and employees of thehealthcare organization. Using one or more machine learning and/or deeplearning techniques, the recommendation component 1602 can identifypatterns in the data attributed to financial and/or clinical gains andlosses and determine recommendation information regarding how to improvefinancial and/or clinical gain and minimize financial and/or clinicalloss. For example, such recommendation information can include theaddition or removal of resources, changes to manners in which thehealthcare organization uses respective equipment, supplies andemployees, and the like. Relevant recommendation information (e.g.,relevant to a current area being viewed by a user and a current contextof the user) can further be selected by the selection component 1016 andprovided to a user as overlay data in an AR or VR experience.

In one or more additional embodiments, the recommendation component 1602can determine recommendation information for providing by a user to aperson (e.g., an employee, a patient or other type of person) that theuser encounters regarding suggested things for the user to say to theperson. The recommendation can be based on a current context of theuser, relevant current information about the other person, and arelationship between the user and the other person. For example, withrespect to an employee, the recommendation component 1602 can determinea suggested thing to say to a particular employee the user encountersthat would improving the clinical and financial performance of thehealthcare organization. According to this example, using one or moremachine learning or deep learning techniques, the recommendationcomponent 1602 can evaluate past performance information associated withan employee and determine an aspect of the employees' past performancethat needs improvement. The recommendation component 1602 can alsodetermine based on the current context of the user and the employee, arelevant suggested remark for the user to say to the employee that wouldencourage the employee to improve his or her performance. The suggestedremark can further be provided to the user as overlay data in AR or VRwhen the user encounters the employee.

For example, in association with a user encountering another employee,the recommendation component 1602 might suggest the user state: “How wasyour birthday last week?”, (e.g., as determined by the recommendationcomponent 1602 based on a birthday in the employee record, comparison tocurrent day, and a determination that the employees birthday was last 6days ago). In another example, in association with a user encounteringphysician at a hospital, the recommendation component 1602 might suggestthe user state: “You had a lot of surgeries yesterday, how'd it go?”,(e.g., as determined by the recommendation component 1602 based onutilization information indicating the physician performed 12 surgeries,comparing this information with the average amount of surgeriesperformed by the physician to determine a 25% higher than normal). Inanother example, in association with a user encountering anotheremployee, the recommendation component 1602 might suggest the userstate: “Is that CT machine in bay 211 still giving you issues?”, (e.g.,based on determinations by the recommendation component 1602 that theuser and the employee are located in bay 211 and that the CT machine hasbeen under service more than a standard amount). Providing “average” or“mean” information against the rest of the healthcare institution orindustry can be incredibly valuable to management and also can providecompetitive incentive for staff to improve to continue to be excellentin their field.

FIG. 17 illustrates a flow diagram of an example method 1700 thatfacilitates providing auxiliary information regarding healthcareprocedure performance using AR in accordance with one or moreembodiments described herein. Repetitive description of like elementsemployed in other embodiments described herein is omitted for sake ofbrevity.

At 1702, a system comprising a processor (e.g., system 100, 600 or 700),receives input data generated in real-time during performance of ahealthcare procedure by a user, wherein the input data comprises videocaptured of the user during the performance of the healthcare relatedprocedure. At 1704, the system, determines descriptive informationcharacterizing the performance based on the input data, wherein thedescriptive information characterizes at least actions performed by theuser during the procedure (e.g., via procedure characterizationcomponent 114). At 1706, the system determines by whether an aspect ofone of the actions currently being performed by the user deviates from adefined protocol for the healthcare related procedure based oncomparison of the descriptive information with reference descriptiveparameters for the healthcare related procedure (e.g., via procedureassessment component 116). At 1708, the system determines feedbackinformation regarding correction of the aspect in response to adetermination that the aspect deviates from the defined protocol (e.g.,via feedback component 120).

FIG. 18 illustrates a flow diagram of another example method 1800 thatfacilitates providing auxiliary information regarding healthcareprocedure performance using AR in accordance with one or moreembodiments described herein. Repetitive description of like elementsemployed in other embodiments described herein is omitted for sake ofbrevity.

At 1802, a system comprising a processor (e.g., system 100, 600 or 700),determines descriptive information regarding performance of a healthcarerelated procedure by a user based on input data generated in real-timeduring the performance of a healthcare procedure by the user, whereinthe input data comprises video captured of the user during theperformance of the healthcare related procedure (e.g., via procedureassessment component 116). At 1804, the system determines whether anaspect of the performance of the healthcare related procedure currentlybeing performed by the user deviates from a defined protocol for thehealthcare related procedure based on comparison of the descriptiveinformation with reference descriptive parameters for the healthcarerelated procedure (e.g., via procedure assessment component 116). At1806, the system determines feedback information regarding correction ofthe aspect in response to a determination that the aspect deviates fromthe defined protocol (e.g., via feedback component 120). At 1808, thesystem generates overlay data representative of the feedback informationfor projection on a display over a current view of the user in responseto the determination that the aspect deviates from the defined protocol(e.g., via AR component 128). At 1810, the system displays the overlaydata on the display in real-time during the performance of thehealthcare related procedure by the user (e.g., via rendering component812).

FIG. 19 illustrates a flow diagram of an example method 1900 thatfacilitates providing auxiliary information regarding healthcare systemperformance in real-time using AR in accordance with one or moreembodiments described herein. Repetitive description of like elementsemployed in other embodiments described herein is omitted for sake ofbrevity.

At 1902, a system comprising a processor (e.g., system 1000, 1300 or1600), analyzes captured image data of a physical area of a healthcarefacility and identifies one or more objects within the captured imagedata (e.g., via environment characterization component 1004). At 1904,the system assesses a state of performance of the healthcare facilitybased on the one or more of the objects (e.g., via environmentassessment component 1006). At 1906, the system determines feedbackinformation regarding the state of performance (e.g., via feedbackcomponent 120). At 1908, the system selects a subset of the feedbackinformation for providing to a user based in part on a context of theuser (e.g., via selection component 1016). At 1910, the system generatesoverlay data representative of the subset of the feedback informationfor projection on a display over a current view of the physical area ofthe healthcare facility that is viewed by the user (e.g., via ARcomponent 128).

In order to provide a context for the various aspects of the disclosedsubject matter, FIG. 20 as well as the following discussion are intendedto provide a general description of a suitable environment in which thevarious aspects of the disclosed subject matter can be implemented. FIG.20 illustrates a block diagram of an example, non-limiting operatingenvironment in which one or more embodiments described herein can befacilitated. Repetitive description of like elements employed in otherembodiments described herein is omitted for sake of brevity.

With reference to FIG. 20, a suitable operating environment 2001 forimplementing various aspects of this disclosure can also include acomputer 2012. The computer 2012 can also include a processing unit2014, a system memory 2016, and a system bus 2018. The system bus 2018couples system components including, but not limited to, the systemmemory 2016 to the processing unit 2014. The processing unit 2014 can beany of various available processors. Dual microprocessors and othermultiprocessor architectures also can be employed as the processing unit2014. The system bus 2018 can be any of several types of busstructure(s) including the memory bus or memory controller, a peripheralbus or external bus, and/or a local bus using any variety of availablebus architectures including, but not limited to, Industrial StandardArchitecture (ISA), Micro-Channel Architecture (MSA), Extended ISA(EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB),Peripheral Component Interconnect (PCI), Card Bus, Universal Serial Bus(USB), Advanced Graphics Port (AGP), Firewire (IEEE 2094), and SmallComputer Systems Interface (SCSI). The system memory 2016 can alsoinclude volatile memory 2020 and nonvolatile memory 2022. The basicinput/output system (BIOS), containing the basic routines to transferinformation between elements within the computer 2012, such as duringstart-up, is stored in nonvolatile memory 2022. By way of illustration,and not limitation, nonvolatile memory 2022 can include read only memory(ROM), programmable ROM (PROM), electrically programmable ROM (EPROM),electrically erasable programmable ROM (EEPROM), flash memory, ornonvolatile random access memory (RAM) (e.g., ferroelectric RAM (FeRAM).Volatile memory 2020 can also include random access memory (RAM), whichacts as external cache memory. By way of illustration and notlimitation, RAM is available in many forms such as static RAM (SRAM),dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM(DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), directRambus RAM (DRRAM), direct Rambus dynamic RAM (DRDRAM), and Rambusdynamic RAM.

Computer 2012 can also include removable/non-removable,volatile/non-volatile computer storage media. FIG. 20 illustrates, forexample, a disk storage 2024. Disk storage 2024 can also include, but isnot limited to, devices like a magnetic disk drive, floppy disk drive,tape drive, Jaz drive, Zip drive, LS-100 drive, flash memory card, ormemory stick. The disk storage 2024 also can include storage mediaseparately or in combination with other storage media including, but notlimited to, an optical disk drive such as a compact disk ROM device(CD-ROM), CD recordable drive (CD-R Drive), CD rewritable drive (CD-RWDrive) or a digital versatile disk ROM drive (DVD-ROM). To facilitateconnection of the disk storage 2024 to the system bus 2018, a removableor non-removable interface is typically used, such as interface 2026.FIG. 20 also depicts software that acts as an intermediary between usersand the basic computer resources described in the suitable operatingenvironment 2001. Such software can also include, for example, anoperating system 2028. Operating system 2028, which can be stored ondisk storage 2024, acts to control and allocate resources of thecomputer 2012. System applications 2030 take advantage of the managementof resources by operating system 2028 through program modules 2032 andprogram data 2034, e.g., stored either in system memory 2016 or on diskstorage 2024. It is to be appreciated that this disclosure can beimplemented with various operating systems or combinations of operatingsystems. A user enters commands or information into the computer 2012through input device(s) 2036. Input devices 2036 include, but are notlimited to, a pointing device such as a mouse, trackball, stylus, touchpad, keyboard, microphone, joystick, game pad, satellite dish, scanner,TV tuner card, digital camera, digital video camera, web camera, and thelike. These and other input devices connect to the processing unit 2014through the system bus 2018 via interface port(s) 2038. Interfaceport(s) 2038 include, for example, a serial port, a parallel port, agame port, and a universal serial bus (USB). Output device(s) 2040 usesome of the same type of ports as input device(s) 2036. Thus, forexample, a USB port can be used to provide input to computer 2012, andto output information from computer 2012 to an output device 2040.Output adapter 2042 is provided to illustrate that there are some outputdevices 2040 like monitors, speakers, and printers, among other outputdevices 2040, which require special adapters. The output adapters 2042include, by way of illustration and not limitation, video and soundcards that provide a means of connection between the output device 2040and the system bus 2018. It should be noted that other devices and/orsystems of devices provide both input and output capabilities such asremote computer(s) 2044.

Computer 2012 can operate in a networked environment using logicalconnections to one or more remote computers, such as remote computer(s)2044. The remote computer(s) 2044 can be a computer, a server, a router,a network PC, a workstation, a microprocessor based appliance, a peerdevice or other common network node and the like, and typically can alsoinclude many or all of the elements described relative to computer 2012.For purposes of brevity, only a memory storage device 2046 isillustrated with remote computer(s) 2044. Remote computer(s) 2044 islogically connected to computer 2012 through a network interface 2048and then physically connected via communication connection 2050. Networkinterface 2048 encompasses wire and/or wireless communication networkssuch as local-area networks (LAN), wide-area networks (WAN), cellularnetworks, etc. LAN technologies include Fiber Distributed Data Interface(FDDI), Copper Distributed Data Interface (CDDI), Ethernet, Token Ringand the like. WAN technologies include, but are not limited to,point-to-point links, circuit switching networks like IntegratedServices Digital Networks (ISDN) and variations thereon, packetswitching networks, and Digital Subscriber Lines (DSL). Communicationconnection(s) 2050 refers to the hardware/software employed to connectthe network interface 2048 to the system bus 2018. While communicationconnection 2050 is shown for illustrative clarity inside computer 2012,it can also be external to computer 2012. The hardware/software forconnection to the network interface 2048 can also include, for exemplarypurposes only, internal and external technologies such as, modemsincluding regular telephone grade modems, cable modems and DSL modems,ISDN adapters, and Ethernet cards.

Embodiments of the present invention may be a system, a method, anapparatus and/or a computer program product at any possible technicaldetail level of integration. The computer program product can include acomputer readable storage medium (or media) having computer readableprogram instructions thereon for causing a processor to carry outaspects of the present invention. The computer readable storage mediumcan be a tangible device that can retain and store instructions for useby an instruction execution device. The computer readable storage mediumcan be, for example, but is not limited to, an electronic storagedevice, a magnetic storage device, an optical storage device, anelectromagnetic storage device, a semiconductor storage device, or anysuitable combination of the foregoing. A non-exhaustive list of morespecific examples of the computer readable storage medium can alsoinclude the following: a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), a static randomaccess memory (SRAM), a portable compact disc read-only memory (CD-ROM),a digital versatile disk (DVD), a memory stick, a floppy disk, amechanically encoded device such as punch-cards or raised structures ina groove having instructions recorded thereon, and any suitablecombination of the foregoing. A computer readable storage medium, asused herein, is not to be construed as being transitory signals per se,such as radio waves or other freely propagating electromagnetic waves,electromagnetic waves propagating through a waveguide or othertransmission media (e.g., light pulses passing through a fiber-opticcable), or electrical signals transmitted through a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network can comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device. Computer readable programinstructions for carrying out operations of various aspects of thepresent invention can be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, configuration data for integrated circuitry, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Smalltalk, C++, or the like, and procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The computer readable program instructions can executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer can be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection can be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) can execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to customize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions. These computer readable programinstructions can be provided to a processor of a general purposecomputer, special purpose computer, or other programmable dataprocessing apparatus to produce a machine, such that the instructions,which execute via the processor of the computer or other programmabledata processing apparatus, create means for implementing thefunctions/acts specified in the flowchart and/or block diagram block orblocks. These computer readable program instructions can also be storedin a computer readable storage medium that can direct a computer, aprogrammable data processing apparatus, and/or other devices to functionin a particular manner, such that the computer readable storage mediumhaving instructions stored therein comprises an article of manufactureincluding instructions which implement aspects of the function/actspecified in the flowchart and/or block diagram block or blocks. Thecomputer readable program instructions can also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational acts to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams can represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the blocks can occur out of theorder noted in the Figures. For example, two blocks shown in successioncan, in fact, be executed substantially concurrently, or the blocks cansometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

While the subject matter has been described above in the general contextof computer-executable instructions of a computer program product thatruns on a computer and/or computers, those skilled in the art willrecognize that this disclosure also can or can be implemented incombination with other program modules. Generally, program modulesinclude routines, programs, components, data structures, etc. thatperform particular tasks and/or implement particular abstract datatypes. Moreover, those skilled in the art will appreciate that theinventive computer-implemented methods can be practiced with othercomputer system configurations, including single-processor ormultiprocessor computer systems, mini-computing devices, mainframecomputers, as well as computers, hand-held computing devices (e.g., PDA,phone), microprocessor-based or programmable consumer or industrialelectronics, and the like. The illustrated aspects can also be practicedin distributed computing environments where tasks are performed byremote processing devices that are linked through a communicationsnetwork. However, some, if not all aspects of this disclosure can bepracticed on stand-alone computers. In a distributed computingenvironment, program modules can be located in both local and remotememory storage devices.

As used in this application, the terms “component,” “system,”“platform,” “interface,” and the like, can refer to and/or can include acomputer-related entity or an entity related to an operational machinewith one or more specific functionalities. The entities disclosed hereincan be either hardware, a combination of hardware and software,software, or software in execution. For example, a component can be, butis not limited to being, a process running on a processor, a processor,an object, an executable, a thread of execution, a program, and/or acomputer. By way of illustration, both an application running on aserver and the server can be a component. One or more components canreside within a process and/or thread of execution and a component canbe localized on one computer and/or distributed between two or morecomputers. In another example, respective components can execute fromvarious computer readable media having various data structures storedthereon. The components can communicate via local and/or remoteprocesses such as in accordance with a signal having one or more datapackets (e.g., data from one component interacting with anothercomponent in a local system, distributed system, and/or across a networksuch as the Internet with other systems via the signal). As anotherexample, a component can be an apparatus with specific functionalityprovided by mechanical parts operated by electric or electroniccircuitry, which is operated by a software or firmware applicationexecuted by a processor. In such a case, the processor can be internalor external to the apparatus and can execute at least a part of thesoftware or firmware application. As yet another example, a componentcan be an apparatus that provides specific functionality throughelectronic components without mechanical parts, wherein the electroniccomponents can include a processor or other means to execute software orfirmware that confers at least in part the functionality of theelectronic components. In an aspect, a component can emulate anelectronic component via a virtual machine, e.g., within a cloudcomputing system.

In addition, the term “or” is intended to mean an inclusive “or” ratherthan an exclusive “or.” That is, unless specified otherwise, or clearfrom context, “X employs A or B” is intended to mean any of the naturalinclusive permutations. That is, if X employs A; X employs B; or Xemploys both A and B, then “X employs A or B” is satisfied under any ofthe foregoing instances. Moreover, articles “a” and “an” as used in thesubject specification and annexed drawings should generally be construedto mean “one or more” unless specified otherwise or clear from contextto be directed to a singular form. As used herein, the terms “example”and/or “exemplary” are utilized to mean serving as an example, instance,or illustration. For the avoidance of doubt, the subject matterdisclosed herein is not limited by such examples. In addition, anyaspect or design described herein as an “example” and/or “exemplary” isnot necessarily to be construed as preferred or advantageous over otheraspects or designs, nor is it meant to preclude equivalent exemplarystructures and techniques known to those of ordinary skill in the art.

As it is employed in the subject specification, the term “processor” canrefer to substantially any computing processing unit or devicecomprising, but not limited to, single-core processors;single-processors with software multithread execution capability;multi-core processors; multi-core processors with software multithreadexecution capability; multi-core processors with hardware multithreadtechnology; parallel platforms; and parallel platforms with distributedshared memory. Additionally, a processor can refer to an integratedcircuit, an application specific integrated circuit (ASIC), a digitalsignal processor (DSP), a field programmable gate array (FPGA), aprogrammable logic controller (PLC), a complex programmable logic device(CPLD), a discrete gate or transistor logic, discrete hardwarecomponents, or any combination thereof designed to perform the functionsdescribed herein. Further, processors can exploit nano-scalearchitectures such as, but not limited to, molecular and quantum-dotbased transistors, switches and gates, in order to optimize space usageor enhance performance of user equipment. A processor can also beimplemented as a combination of computing processing units. In thisdisclosure, terms such as “store,” “storage,” “data store,” datastorage,” “database,” and substantially any other information storagecomponent relevant to operation and functionality of a component areutilized to refer to “memory components,” entities embodied in a“memory,” or components comprising a memory. It is to be appreciatedthat memory and/or memory components described herein can be eithervolatile memory or nonvolatile memory, or can include both volatile andnonvolatile memory. By way of illustration, and not limitation,nonvolatile memory can include read only memory (ROM), programmable ROM(PROM), electrically programmable ROM (EPROM), electrically erasable ROM(EEPROM), flash memory, or nonvolatile random access memory (RAM) (e.g.,ferroelectric RAM (FeRAM). Volatile memory can include RAM, which canact as external cache memory, for example. By way of illustration andnot limitation, RAM is available in many forms such as synchronous RAM(SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rateSDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM),direct Rambus RAM (DRRAM), direct Rambus dynamic RAM (DRDRAM), andRambus dynamic RAM (RDRAM). Additionally, the disclosed memorycomponents of systems or computer-implemented methods herein areintended to include, without being limited to including, these and anyother suitable types of memory.

What has been described above include mere examples of systems andcomputer-implemented methods. It is, of course, not possible to describeevery conceivable combination of components or computer-implementedmethods for purposes of describing this disclosure, but one of ordinaryskill in the art can recognize that many further combinations andpermutations of this disclosure are possible. Furthermore, to the extentthat the terms “includes,” “has,” “possesses,” and the like are used inthe detailed description, claims, appendices and drawings such terms areintended to be inclusive in a manner similar to the term “comprising” as“comprising” is interpreted when employed as a transitional word in aclaim. The descriptions of the various embodiments have been presentedfor purposes of illustration, but are not intended to be exhaustive orlimited to the embodiments disclosed. Many modifications and variationswill be apparent to those of ordinary skill in the art without departingfrom the scope and spirit of the described embodiments. The terminologyused herein was chosen to best explain the principles of theembodiments, the practical application or technical improvement overtechnologies found in the marketplace, or to enable others of ordinaryskill in the art to understand the embodiments disclosed herein.

What is claimed is:
 1. A system, comprising: a memory that storescomputer executable components; a processor that executes computerexecutable components stored in the memory, wherein the computerexecutable components comprise: an environment recognition componentconfigured to identify an area of a healthcare facility included inimage data captured of the healthcare facility; an environmentcharacterization component configured to characterize aspects of acurrent state of the area of the area based on analysis of the imagedata, including determining object information characterizing objectsincluded in the area; an environment assessment component configured todetermine whether one or more of the aspects of the current state of thearea deviate from a defined requirement for the area based on evaluationof the object information; and a feedback component configured togenerate feedback information regarding an aspect of the current stateof the area that deviates from the defined requirement for the area inresponse to a determination, by the environment assessment component,that the aspect of the of the current state of the area deviates fromthe defined requirement for the area.
 2. The system of claim 1, furthercomprising: an augmented reality data component configured to generateoverlay data comprising the feedback information for projection on adisplay of a device over a current view of the area as viewed by a useron or through the display.
 3. The system of claim 2, wherein the imagedata comprises one or more images captured via a camera of the device.4. The system of claim 2, further comprising: a rendering componentconfigured to present the overlay data on the display.
 5. The system ofclaim 4, wherein the display is integrated into an eyewear device wornby the user.
 6. The system of claim 1, wherein the object informationidentifies the objects included in the area and wherein the environmentassessment component is configured to determine whether the one or moreof the aspects of the current state of the area deviate from the definedrequirement for the area based on comparison of the objects included inthe area with reference information identifying defined objects thatshould be included in the area or shouldn't be included in the area. 7.The system of claim 6, wherein the determination that the aspect of theof the current state of the area deviates from the defined requirementfor the area is based on recognition, in response to the comparison, ofan object included in the area that shouldn't be included in the area.8. The system of claim 6, wherein the determination that the aspect ofthe of the current state of the area deviates from the definedrequirement for the area is based on recognition, in response to thecomparison, of a defined object that should be included in the area thatis not included in the area.
 9. The system of claim 1, wherein theobject information characterizes physical states of the objects includedin the area and wherein the environment assessment component isconfigured to determine whether the one or more of the aspects of thecurrent state of the area deviate from the defined requirement for thearea based on comparison of the physical states of the objects withreference information identifying defined physical states for theobjects.
 10. The system of claim 1, wherein the object informationidentifies relative spatial positions of the objects included in thearea and wherein the environment assessment component is configured todetermine whether the one or more of the aspects of the current state ofthe area deviate from the defined requirement for the area based oncomparison of the relative spatial positions of the objects withreference information identifying defined spatial positions for theobjects as included in the area.
 11. The system of claim 10, wherein thearea comprises an operating room in which a medical operation is beingperformed and the objects comprise a medical instrument and a patient,and wherein the determination that the aspect of the of the currentstate of the area deviates from the defined requirement for the area isbased on recognition, in response to the comparison, that a spatialposition of the medical instrument relative to the patient deviates froma defined spatial position for the medical instrument relative to thepatient.
 12. The system of claim 1, wherein the objects comprise medicalequipment or supplies.
 13. A method comprising: identifying, by a systemcomprising a processor, an area of a healthcare facility included inimage data captured of the healthcare facility; characterizing, by thesystem, aspects of a current state of the area of the area based onanalysis of the image data, including determining object informationcharacterizing objects included in the area; determining, by the system,whether one or more of the aspects of the current state of the areadeviate from a defined requirement for the area based on evaluation ofthe object information; and based on a determination that an aspect ofthe current state of the area deviates from the defined requirement forthe area, generating, by the system, feedback information regarding theaspect of the current state of the area that deviates from the definedrequirement for the area.
 14. The method of claim 13, wherein thegenerating the feedback information comprises generating overlay datacomprising the feedback information for projection on a display of adevice over a current view of the area as viewed by a user on or throughthe display.
 15. The method of claim 14, further comprising: rendering,by the system, the overlay data on the display.
 16. The method of claim13, wherein the determining the object information comprises determiningidentification information identifying the objects included in the area,and wherein the determining whether the one or more of the aspects ofthe current state of the area deviate from the defined requirement forthe area comprises comparing the identification information withreference identification information identifying defined objects thatshould be included in the area or shouldn't be included in the area. 17.The system of claim 1, wherein the determining the object informationcomprises determining state information identifying states of theobjects included in the area, and wherein the determining whether theone or more of the aspects of the current state of the area deviate fromthe defined requirement for the area comprises comparing the stateinformation with reference state information identifying defined statesfor the objects as included in the area.
 18. The system of claim 1,wherein the determining the object information comprises determiningposition information identifying relative positions of the objectsincluded in the area, and wherein the determining whether the one ormore of the aspects of the current state of the area deviate from thedefined requirement for the area comprises comparing the positioninformation with reference position information identifying definedrelative positions for the objects as included in the area.
 19. Amachine-readable storage medium, comprising executable instructionsthat, when executed by a processor, facilitate performance ofoperations, comprising: receiving image data captured of an area of ahealthcare facility; characterizing aspects of a current state of thearea of the area based on analysis of the image data, includingdetermining object information identifying one or more characteristicsof objects included in the area; determining whether one or more of theaspects of the current state of the area deviate from a definedrequirement for the area based on comparing the object information withreference object information regarding reference characteristics fordefined objects as included in the area; and based on a determinationthat an aspect of the current state of the area deviates from thedefined requirement for the area, generating feedback informationregarding the aspect of the current state of the area that deviates fromthe defined requirement for the area.
 20. The machine-readable storagemedium of claim 19, wherein the generating the feedback informationcomprises generating overlay data comprising the feedback informationfor projection on a display of a device over a current view of the areaas viewed by a user on or through the display.